when i training it in this code:
python -m torch.distributed.launch --nproc_per_node=8
metro/tools/run_metro_bodymesh.py
--train_yaml Tax-H36m-coco40k-Muco-UP-Mpii/train.yaml
--val_yaml human3.6m/valid.protocol2.yaml
--arch hrnet-w64
--num_workers 4
--per_gpu_train_batch_size 30
--per_gpu_eval_batch_size 30
--num_hidden_layers 4
--num_attention_heads 4
--lr 1e-4
--num_train_epochs 200
--input_feat_dim 2051,512,128
--hidden_feat_dim 1024,256,128

then it will happen error like that:

=======================================
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "metro/tools/run_metro_bodymesh.py", line 714, in
Traceback (most recent call last):
File "metro/tools/run_metro_bodymesh.py", line 714, in
File "metro/tools/run_metro_bodymesh.py", line 714, in
File "metro/tools/run_metro_bodymesh.py", line 714, in
Traceback (most recent call last):
Traceback (most recent call last):
File "metro/tools/run_metro_bodymesh.py", line 714, in
File "metro/tools/run_metro_bodymesh.py", line 714, in
main(args)
File "metro/tools/run_metro_bodymesh.py", line 587, in main
Traceback (most recent call last):
main(args)
main(args)
main(args)
File "metro/tools/run_metro_bodymesh.py", line 714, in
File "metro/tools/run_metro_bodymesh.py", line 587, in main
File "metro/tools/run_metro_bodymesh.py", line 587, in main
Traceback (most recent call last):
File "metro/tools/run_metro_bodymesh.py", line 587, in main
main(args)
main(args)
File "metro/tools/run_metro_bodymesh.py", line 714, in
File "metro/tools/run_metro_bodymesh.py", line 587, in main
File "metro/tools/run_metro_bodymesh.py", line 587, in main
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
main(args)
File "metro/tools/run_metro_bodymesh.py", line 587, in main
main(args)
File "metro/tools/run_metro_bodymesh.py", line 587, in main
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
File "/usr/local/lib/python3.6/os.py", line 669, in getitem
raise KeyError(key) from None
raise KeyError(key) from None
raise KeyError(key) from None
raise KeyError(key) from None
raise KeyError(key) from None
raise KeyError(key) from None
raise KeyError(key) from None
KeyError: 'NODE_RANK'
raise KeyError(key) from None
KeyError: 'NODE_RANK'
KeyError: 'NODE_RANK'
KeyError: 'NODE_RANK'
KeyError: 'NODE_RANK'
KeyError: 'NODE_RANK'
KeyError: 'NODE_RANK'
KeyError: 'NODE_RANK'
Traceback (most recent call last):
File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in
main()
File "/usr/local/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-u', 'metro/tools/run_metro_bodymesh.py', '--local_rank=7', '--train_yaml', 'Tax-H36m-coco40k-Muco-UP-Mpii/train.yaml', '--val_yaml', 'human3.6m/valid.protocol2.yaml', '--arch', 'hrnet-w64', '--num_workers', '4', '--per_gpu_train_batch_size', '30', '--per_gpu_eval_batch_size', '30', '--num_hidden_layers', '4', '--num_attention_heads', '4', '--lr', '1e-4', '--num_train_epochs', '200', '--input_feat_dim', '2051,512,128', '--hidden_feat_dim', '1024,256,128']' returned non-zero exit status 1.**

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

how can i deal with it?

Hi,

thanks for the amazing work.

I have a question regarding some initial output results (as the training seems to be taking, the next, 40 days on my single GTX 3090 card).

As one can see in the image bellow there are 2d keypoints in the left column, as well as the estimated SMPL in the right one.

There are examples of fitting in the places where it, I assume intentionally, occluded. Is it a potential issue?

Also in another example, there is a weird SMPL estimation, and there are plenty of those, in some batches it estimates without these spikes.

Any feedback on those are highly appreciated.

Cheers!

Hello,
thank you very much for this great project. I have done many tests (inference only). The results are very bad and I want to ask you if I am doing something wrong? I did exactly what is suggested in DEMO.md. Should I do something additional?
Here some poor results:

Hi!

I found that a subset of MSCOCO 2017 dataset is used instead of the original dataset. Why you choose these 40k samples? What do you go by when pick them out? Have you tried using the full MSCOCO dataset to train and is there any difference?

I guess it's to guarantee at least several valid keypoints, but I wish a confirm.

Look forward to your reply!

Hello,

When I try to download the data from 'https://datarelease.blob.core.windows.net/metro/datasets/filename.tar' using the following command:

%%bash
# copy Snapshot Serengeti images to a local directory
DATASET_URL="https://datarelease.blob.core.windows.net/metro/datasets/filename.tar"
LOCAL_DIR="/content/MeshTransformer"

./azcopy cp "${DATASET_URL}" "${LOCAL_DIR}" 

I get this error:

INFO: Scanning...

failed to perform copy command due to error: Login Credentials missing. No SAS token or OAuth token is present and the resource is not public

Is it possible to download the data?

As you metioned in EXP.md.

python metro/tools/run_hand_inference.py
--model_path models/metro_released/metro_hand_state_dict.bin \

But there is no this file.

Hi, thanks for the excellent work.

I want to ask about how you set the focal length of a mesh renderer.
In the demo codes, you set it to 1000 and it works well on the sample FreiHAND images.
However, when I test on other images, I have to set the focal length to 5000 for good projection.
Is there any math or reason behind setting the focal length? Thanks.

And by the way, the model does not perform bad on left hands as you've written in the demo README.
You can flip the input image of a left hand and assume it as a right hand.

Hi,

It is asking for
ileNotFoundError: [Errno 2] No such file or directory: 'models/hrnet/cls_hrnet_w64_sgd_lr5e-2_wd1e-4_bs32_x100.yaml'
when running
python ./metro/tools/end2end_inference_bodymesh.py --resume_checkpoint ./models/metro_release/metro_3dpw_state_dict.bin --image_file_or_path ./samples/human-body

There is no models/hrnet/ folder . Where should it be ?

Could anyone please fix the following error? Thanks a lot.

"/usr/local/lib/python3.7/dist-packages/mesh_transformer/util.py in ()
34
35
---> 36 class ClipByGlobalNormState(OptState):
37 """The clip_by_global_norm transformation is stateless."""
38

/usr/lib/python3.7/typing.py in new(cls, *args, **kwds)
308 isinstance(args[1], tuple)):
309 # Close enough.
--> 310 raise TypeError(f"Cannot subclass {cls!r}")
311 return super().new(cls)
312

TypeError: Cannot subclass <class 'typing._SpecialForm'>"

Hello. Thanks for your great work. I have some questions about training on 3DPW.
According to docs/EXP.md, i fine-tune your pre-trained model (metro_h36m_state_dict.bin) on 3DPW training set. But the result is not like what you described in the paper. How did you train on the 3DPW data set?

Hello,

Thank you for your wonderful work. It's really impressive.

let's cut to the chase, how to fine-tone your pre-train model using my data. Specifically, how to reorganize my data format (.png) to the following:

| |-- up3d
| | |-- trainval.img.tsv
| | |-- trainval.hw.tsv
| | |-- trainval.label.tsv
| | |-- trainval.linelist.tsv

I would appreciate any and all help you can give!

Thanks.

Hello,

Thank you for publishing such a complete project for your work. It is very much appreciated!

Is there a way to figure out the correspondence between the person id and the timestep for video datasets? For example, I want to edit the dataloader to randomly sample image crops of the same person in chronological order e.g. [person i @ frame 0 in video k, person i @ frame 1 in video k, person i @ frame 2 in video k, etc. ] where i and k are randomly sampled.

Is this possible given the meta data for 3DPW, Human 3.6 Million, or any of the other video based datasets? Or would I have to generate my own .tsv files? I would appreciate any and all help you can give!

Thanks

To deal with multi-person 3d pose estimation, I've run the METRO to get 3d joints from smpl (use get_joints function) on the cropped patches of image. So now I have the joints with shape 1x24x3 for each patch. For a given cropped bounding box like [x, y, w, h]. How can I reverse to the original image 3d joints? Thanks

Hey, congrats for your amazing work..i would like to ask about your experiments that you replaced the template mesh encodings with the sinusoidal ones..can you provide the code that you used to concatenate the image tokens with the sinusoidal positions? Thanks in advance

Hi,

Thank you for the great work! In the results section of your paper, you've stated results for your training on many datasets for 200 epochs. I was wondering if you also happen to have trained your model only on Human3.6M or Human3.6M+COCO. If you do, I'd be grateful if you could let me know.

Thanks in advance

Hi, thank you for open source your nice work.
I was wondering about the number of supported sequence lengths.
Because when I use the sequence length larger than 512, it throws the following error(the code is ok with a smaller sequence length with default setting):

/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [19,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [20,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [21,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [22,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [23,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [24,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [25,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [26,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [27,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [28,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [29,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [30,0,0] Assertion srcIndex < srcSelectDimSize failed.
/tmp/pip-req-build-g2m34a_4/aten/src/ATen/native/cuda/Indexing.cu:690: indexSelectLargeIndex: block: [64,0,0], thread: [31,0,0] Assertion srcIndex < srcSelectDimSize failed.

RuntimeError: CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below
might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. CUDA error: device-side assert triggered

How to get 3D coordinates of camera coordinate system instead of MANO space？

Considering that the absolute coordinates of hand are not obtained, how can 3d hand tracking be applied to augmented reality scenes such as AR glasses?

subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-u', 'metro/tools/run_metro_bodymesh.py', '--local_rank=3', '--train_yaml', 'Tax-H36m-coco40k-Muco-UP-Mpii/train.yaml', '--val_yaml', 'human3.6m/valid.protocol2.yaml', '--arch', 'hrnet-w64', '--num_workers', '4', '--per_gpu_train_batch_size', '30', '--per_gpu_eval_batch_size', '30', '--num_hidden_layers', '4', '--num_attention_heads', '4', '--lr', '1e-4', '--num_train_epochs', '200', '--input_feat_dim', '2051,512,128', '--hidden_feat_dim', '1024,256,128']' returned non-zero exit status 1.

Hi!
I have some confusion about the mappings of image features to vertex embeddings.

After reading the paper, I thought the image feature vector of depth 2048 is expanded to the number of vertices and joints, to get vertex embeddings of shape (V + J) * 2048, before feeding it to the transformer encoder. GraphCMR also do the work like this. The codes of HandMesh model in metro/modeling/bert/modeling_metro.py at Line 182~185 seems to confirm my assumption:

# extract global image feature using a CNN backbone
image_feat = self.backbone(images)

# concatinate image feat and template mesh
image_feat = image_feat.view(batch_size, 1, 2048).expand(-1, ref_vertices.shape[-2], -1)
features = torch.cat([ref_vertices, image_feat], dim=2)

and of course, the backbone here should drop the last layer which is usually used to further map the dimension 2048 to #classifications, so as to retain feature vector of dimention 2048. This is confirmed in metro/tools/run_metro_handmesh.py#L646 at Line 646:

# remove the last fc layer
backbone = torch.nn.Sequential(*list(backbone.children())[:-1])

However, BodyMesh model uses a different strategy (in metro/modeling/bert/modeling_metro.py#L265 at Line 265~270):

# extract image feature maps using a CNN backbone
image_feat = self.backbone(images)
image_feat_newview = image_feat.view(batch_size,2048,-1)
image_feat_newview = image_feat_newview.transpose(1,2)
# and apply a conv layer to learn image token for each 3d joint/vertex position
img_tokens = self.conv_learn_tokens(image_feat_newview)

the definition of self.conv_learn_tokens is:

self.conv_learn_tokens = torch.nn.Conv1d(49,431+14,1)

so, clearly, here the backbone outputs a feature map of shape 2048 * 7 * 7. After flattens it to 2048 * 49, a conv1d layer conv_learn_tokens (could be seen as a linear layer) map the 2D image features to vertex embeddings (49 -> 431+14).
Here the backbone network should drop the last 2 layers, including AvgPool (metro/tools/run_metro_bodymesh.py#L675):

# remove the last fc layer
backbone = torch.nn.Sequential(*list(backbone.children())[:-2])

So: Why you use 2 different strategies to deal with the mapping (from image features to vertex embeddings)? Is there something to do with the differences of Human Mesh Reconstruction and Hand Mesh Reconstruction?

Looking forward for your reply, that'll help a lot.

Hi, thanks for your great work, and I notice that the code can get pred_3d_joints and pred_3d_joints_from_smpl, when I print their shape, I got 14 and 17 respectively, but since smpl have 24 joints, hope you can tell which joints are contained in those predicted joints,
And I also hope if you can give some advice on how to obtain the root motion based on the predicted, thanks~

Thanks for your releasing! When I run run_metro_bodymesh.py using 3DPW data, I meet a problem. It occurs in modeling_metro.py. I search the source and find it has wrong in row 437, layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i]) inmodeling_bert.py. Specificially, it reports head_mask[i]=None. As a newer in transformer, I have no idea where the head_mask is difined in your code. Could you help me about my confusion?

Hi, thanks for your great work. When I install OpenDR, here is the error. Do you know how to solve this?

(metro) cezheng@lambda-dual:~/HPE/meshpose/MeshTransformer/apex$ pip install opendr matplotlib
Collecting opendr
Using cached opendr-0.78.tar.gz (581 kB)
Requirement already satisfied: matplotlib in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (3.3.2)
Requirement already satisfied: Cython in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from opendr) (0.29.21)
Requirement already satisfied: chumpy>=0.58 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from opendr) (0.69)
Requirement already satisfied: pillow>=6.2.0 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (7.2.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (1.2.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: numpy>=1.15 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (1.19.2)
Requirement already satisfied: certifi>=2020.06.20 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (2021.5.30)
Requirement already satisfied: python-dateutil>=2.1 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (2.8.1)
Requirement already satisfied: cycler>=0.10 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: scipy>=0.13.0 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from chumpy>=0.58->opendr) (1.6.2)
Requirement already satisfied: six>=1.11.0 in /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages (from chumpy>=0.58->opendr) (1.15.0)
Building wheels for collected packages: opendr
Building wheel for opendr (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/cezheng/anaconda3/envs/metro/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kws8g40y/opendr/setup.py'"'"'; file='"'"'/tmp/pip-install-kws8g40y/opendr/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-j7sa3lsx
cwd: /tmp/pip-install-kws8g40y/opendr/
Complete output (72 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/opendr
copying opendr/test_depth_renderer.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/occlusion_test.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/topology.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/lighting.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/everything.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/utils.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/dummy.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/version.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/camera.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/init.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/simple.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_renderer.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/util_tests.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/serialization.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_sh.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/slider_demo.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/geometry.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/filters.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/cvwrap.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_geometry.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_camera.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/common.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/renderer.py -> build/lib.linux-x86_64-3.8/opendr
creating build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/draw_triangle_shaders_2_1.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/init.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/constants.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/autogen.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/fix_warnings.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/draw_triangle_shaders_3_2.py -> build/lib.linux-x86_64-3.8/opendr/contexts
creating build/lib.linux-x86_64-3.8/opendr/test_dr
copying opendr/test_dr/init.py -> build/lib.linux-x86_64-3.8/opendr/test_dr
running build_ext
building 'opendr.contexts.ctx_mesa' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/opendr
creating build/temp.linux-x86_64-3.8/opendr/contexts
gcc -pthread -B /home/cezheng/anaconda3/envs/metro/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D__OSMESA_=1 -Iopendr/contexts -I. -I/home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include -Iopendr/contexts/OSMesa/include -I/home/cezheng/anaconda3/envs/metro/include/python3.8 -c opendr/contexts/ctx_mesa.c -o build/temp.linux-x86_64-3.8/opendr/contexts/ctx_mesa.o -lstdc++
In file included from /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
from /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from opendr/contexts/ctx_mesa.c:657:
/home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with "
| ^~~~~~~
In file included from opendr/contexts/ctx_mesa.c:663:
opendr/contexts/OSMesa/include/GL/osmesa.h:258:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
258 | typedef void (*OSMESAproc)();
| ^~~~~~~
opendr/contexts/ctx_mesa.c: In function ‘__pyx_pf_6opendr_8contexts_8ctx_mesa_13OsContextBase_150ShaderSource’:
opendr/contexts/ctx_mesa.c:13123:50: warning: passing argument 3 of ‘glShaderSource’ from incompatible pointer type [-Wincompatible-pointer-types]
13123 | glShaderSource(__pyx_v_shader, __pyx_v_count, (&__pyx_v_s), (&__pyx_v_len));
| ~^~~~~~~~~~~
| |
| char **
In file included from opendr/contexts/OSMesa/include/GL/gl.h:2085,
from opendr/contexts/gl_includes.h:10,
from opendr/contexts/ctx_mesa.c:662:
opendr/contexts/OSMesa/include/GL/glext.h:5794:82: note: expected ‘const GLchar **’ {aka ‘const char **’} but argument is of type ‘char *’
5794 | GLAPI void APIENTRY glShaderSource (GLuint shader, GLsizei count, const GLchar *string, const GLint *length);
| ~~~~~~~~~~~~~~~^~~~~~
gcc -pthread -shared -B /home/cezheng/anaconda3/envs/metro/compiler_compat -L/home/cezheng/anaconda3/envs/metro/lib -Wl,-rpath=/home/cezheng/anaconda3/envs/metro/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/opendr/contexts/ctx_mesa.o -Lopendr/contexts/OSMesa/lib -lOSMesa -lGL -lGLU -o build/lib.linux-x86_64-3.8/opendr/contexts/ctx_mesa.cpython-38-x86_64-linux-gnu.so -lstdc++
/home/cezheng/anaconda3/envs/metro/compiler_compat/ld: cannot find -lOSMesa
/home/cezheng/anaconda3/envs/metro/compiler_compat/ld: cannot find -lGLU
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

ERROR: Failed building wheel for opendr
Running setup.py clean for opendr
Failed to build opendr
DEPRECATION: Could not build wheels for opendr which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at pypa/pip#8368.
Installing collected packages: opendr
Running setup.py install for opendr ... error
ERROR: Command errored out with exit status 1:
command: /home/cezheng/anaconda3/envs/metro/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kws8g40y/opendr/setup.py'"'"'; file='"'"'/tmp/pip-install-kws8g40y/opendr/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-2a0woggk/install-record.txt --single-version-externally-managed --compile --install-headers /home/cezheng/anaconda3/envs/metro/include/python3.8/opendr
cwd: /tmp/pip-install-kws8g40y/opendr/
Complete output (72 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/opendr
copying opendr/test_depth_renderer.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/occlusion_test.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/topology.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/lighting.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/everything.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/utils.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/dummy.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/version.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/camera.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/init.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/simple.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_renderer.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/util_tests.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/serialization.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_sh.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/slider_demo.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/geometry.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/filters.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/cvwrap.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_geometry.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/test_camera.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/common.py -> build/lib.linux-x86_64-3.8/opendr
copying opendr/renderer.py -> build/lib.linux-x86_64-3.8/opendr
creating build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/draw_triangle_shaders_2_1.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/init.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/constants.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/autogen.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/fix_warnings.py -> build/lib.linux-x86_64-3.8/opendr/contexts
copying opendr/contexts/draw_triangle_shaders_3_2.py -> build/lib.linux-x86_64-3.8/opendr/contexts
creating build/lib.linux-x86_64-3.8/opendr/test_dr
copying opendr/test_dr/init.py -> build/lib.linux-x86_64-3.8/opendr/test_dr
running build_ext
building 'opendr.contexts.ctx_mesa' extension
creating build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/opendr
creating build/temp.linux-x86_64-3.8/opendr/contexts
gcc -pthread -B /home/cezheng/anaconda3/envs/metro/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -D__OSMESA_=1 -Iopendr/contexts -I. -I/home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include -Iopendr/contexts/OSMesa/include -I/home/cezheng/anaconda3/envs/metro/include/python3.8 -c opendr/contexts/ctx_mesa.c -o build/temp.linux-x86_64-3.8/opendr/contexts/ctx_mesa.o -lstdc++
In file included from /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
from /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from opendr/contexts/ctx_mesa.c:657:
/home/cezheng/anaconda3/envs/metro/lib/python3.8/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with "
| ^~~~~~~
In file included from opendr/contexts/ctx_mesa.c:663:
opendr/contexts/OSMesa/include/GL/osmesa.h:258:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
258 | typedef void (*OSMESAproc)();
| ^~~~~~~
opendr/contexts/ctx_mesa.c: In function ‘__pyx_pf_6opendr_8contexts_8ctx_mesa_13OsContextBase_150ShaderSource’:
opendr/contexts/ctx_mesa.c:13123:50: warning: passing argument 3 of ‘glShaderSource’ from incompatible pointer type [-Wincompatible-pointer-types]
13123 | glShaderSource(__pyx_v_shader, __pyx_v_count, (&__pyx_v_s), (&__pyx_v_len));
| ~^~~~~~~~~~~
| |
| char **
In file included from opendr/contexts/OSMesa/include/GL/gl.h:2085,
from opendr/contexts/gl_includes.h:10,
from opendr/contexts/ctx_mesa.c:662:
opendr/contexts/OSMesa/include/GL/glext.h:5794:82: note: expected ‘const GLchar **’ {aka ‘const char **’} but argument is of type ‘char *’
5794 | GLAPI void APIENTRY glShaderSource (GLuint shader, GLsizei count, const GLchar *string, const GLint *length);
| ~~~~~~~~~~~~~~~^~~~~~
gcc -pthread -shared -B /home/cezheng/anaconda3/envs/metro/compiler_compat -L/home/cezheng/anaconda3/envs/metro/lib -Wl,-rpath=/home/cezheng/anaconda3/envs/metro/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.8/opendr/contexts/ctx_mesa.o -Lopendr/contexts/OSMesa/lib -lOSMesa -lGL -lGLU -o build/lib.linux-x86_64-3.8/opendr/contexts/ctx_mesa.cpython-38-x86_64-linux-gnu.so -lstdc++
/home/cezheng/anaconda3/envs/metro/compiler_compat/ld: cannot find -lOSMesa
/home/cezheng/anaconda3/envs/metro/compiler_compat/ld: cannot find -lGLU
collect2: error: ld returned 1 exit status
error: command 'gcc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/cezheng/anaconda3/envs/metro/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-kws8g40y/opendr/setup.py'"'"'; file='"'"'/tmp/pip-install-kws8g40y/opendr/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-2a0woggk/install-record.txt --single-version-externally-managed --compile --install-headers /home/cezheng/anaconda3/envs/metro/include/python3.8/opendr Check the logs for full command output.

Hi authors,

I am facing the problem when training the model using:

python -m torch.distributed.launch --nproc_per_node=8 \
       metro/tools/run_metro_bodymesh.py \
       --train_yaml Tax-H36m-coco40k-Muco-UP-Mpii/train.yaml \
       --val_yaml human3.6m/valid.protocol2.yaml \
       --arch hrnet-w64 \
       --num_workers 4 \
       --per_gpu_train_batch_size 4 \
       --per_gpu_eval_batch_size 4 \
       --num_hidden_layers 4 \
       --num_attention_heads 4 \
       --lr 1e-4 \
       --num_train_epochs 200 \
       --input_feat_dim 2051,512,128 \
       --hidden_feat_dim 1024,256,128

I met the error related to NODE_RANK. The log is as follows:

*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
*****************************************
Traceback (most recent call last):
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
        main(args)main(args)

  File "metro/tools/run_metro_bodymesh.py", line 587, in main
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
    main(args)
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
        print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))

  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
    print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
        raise KeyError(key) from Noneraise KeyError(key) from None

KeyErrorKeyError: 'NODE_RANK':
'NODE_RANK'
    raise KeyError(key) from None
KeyError: 'NODE_RANK'
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
    main(args)
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
    print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'NODE_RANK'
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
    main(args)
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
    print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
    main(args)
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
    raise KeyError(key) from None
KeyError: 'NODE_RANK'
    print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'NODE_RANK'
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
    main(args)
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
    print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'NODE_RANK'
Traceback (most recent call last):
  File "metro/tools/run_metro_bodymesh.py", line 714, in <module>
    main(args)
  File "metro/tools/run_metro_bodymesh.py", line 587, in main
    print("Init distributed training on local rank {} ({}), rank {}, world size {}".format(args.local_rank, int(os.environ["LOCAL_RANK"]), int(os.environ["NODE_RANK"]), args.num_gpus))
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None
KeyError: 'NODE_RANK'
Killing subprocess 22973
Killing subprocess 22974
Killing subprocess 22975
Killing subprocess 22976
Killing subprocess 22977
Killing subprocess 22978
Killing subprocess 22979
Killing subprocess 22980
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in <module>
    main()
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
    sigkill_handler(signal.SIGTERM, None)  # not coming back
  File "/home/ubuntu/anaconda3/envs/metro/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
    raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ubuntu/anaconda3/envs/metro/bin/python', '-u', 'metro/tools/run_metro_bodymesh.py', '--local_rank=7', '--train_yaml', 'Tax-H36m-coco40k-Muco-UP-Mpii/train.yaml', '--val_yaml', 'human3.6m/valid.protocol2.yaml', '--arch', 'hrnet-w64', '--num_workers', '4', '--per_gpu_train_batch_size', '4', '--per_gpu_eval_batch_size', '4', '--num_hidden_layers', '4', '--num_attention_heads', '4', '--lr', '1e-4', '--num_train_epochs', '200', '--input_feat_dim', '2051,512,128', '--hidden_feat_dim', '1024,256,128']' returned non-zero exit status 1.

Could you please give me some solutions?
Thank you in advance!

Hi, how to solve the error when i run end2end_inference_bodymesh.py?

config.output_attentions = False
AttributeError: 'NoneType' object has no attribute 'output_attentions'

Greetings,

I couldn't understand why trans parameter is not passed to the mano layer for extracting gt_3d_joints and gt_vertices. Some of the implementations (I2L-MeshNet) were using mano_trans and I'm confused about its function.

Besides, I'm confused about subtracting the root wrist from gt_vertices and 'gt_3d_joints'. Why do you perform this operation in both training and inference?

Thanks in advance.

Hi,

Delving into your code, I see that you were using PAmPJPE for selecting the better model.

Is there any intuition for that?

Thank you!

Hello. I met an error when I am running the demo.

Traceback (most recent call last):
File "./metro/tools/end2end_inference_bodymesh.py", line 318, in
main(args)
File "./metro/tools/end2end_inference_bodymesh.py", line 314, in main
run_inference(args, image_list, _metro_network, mesh_smpl, renderer, mesh_sampler)
File "./metro/tools/end2end_inference_bodymesh.py", line 90, in run_inference
att[-1][0].detach())
File "./metro/tools/end2end_inference_bodymesh.py", line 121, in visualize_mesh_and_attention
rend_img = visualize_reconstruction_and_att_local(img, 224, vertices_full, vertices, vertices_2d, cam, renderer, joints_2d, att, color='pink')
File "/home/user/Desktop/MeshTransformer/metro/utils/renderer.py", line 407, in visualize_reconstruction_and_att_local
focal_length=focal_length, body_color=color)
File "/home/user/Desktop/MeshTransformer/metro/utils/renderer.py", line 608, in render
return self.renderer.r
File "/home/user/anaconda3/envs/metro/lib/python3.7/site-packages/chumpy/ch.py", line 594, in r
self._call_on_changed()
File "/home/user/anaconda3/envs/metro/lib/python3.7/site-packages/chumpy/ch.py", line 589, in _call_on_changed
self.on_changed(self._dirty_vars)
File "/home/user/anaconda3/envs/metro/lib/python3.7/site-packages/opendr-0.73-py3.7.egg/opendr/renderer.py", line 1082, in on_changed
self.vbo_verts_face.set_array(np.array(self.verts_by_face).astype(np.float32))
AttributeError: 'ColoredRenderer' object has no attribute 'vbo_verts_face'

The following message showed after run the human body reconstruction script of Quick Demo.

Traceback (most recent call last):
  File "./metro/tools/end2end_inference_bodymesh.py", line 28, in <module>
    from metro.utils.renderer import Renderer, visualize_reconstruction, visualize_reconstruction_test, visualize_reconstruction_no_text, visualize_reconstruction_and_att_local
  File "/root/MeshTransformer/metro/utils/renderer.py", line 15, in <module>
    from opendr.renderer import ColoredRenderer, TexturedRenderer
  File "/root/anaconda3/envs/maskrcnn/lib/python3.7/site-packages/opendr/renderer.py", line 25, in <module>
    from .contexts.ctx_mesa import OsContext
  File "opendr/contexts/ctx_base.pyx", line 18, in init opendr.contexts.ctx_mesa
ModuleNotFoundError: No module named '_constants'

And I have no clue how to fix it.
If anyone could come up with a solution. I really appreciate it.

Hi, I'm curious about the quantitative performance of METERO with ResNet50 backbone on the 3DPW dataset, since the official repo doesn't provide the pre-trained models with ResNet50. I'd be grateful if any advice was given.

I am unable to get the results on the human3.6m dataset as published in the paper. Im getting 55.1584 MPJPE and 39.2701 PA-MPJPE on models/metro_release/metro_h36m_state_dict.bin

I am loading the model in the same way as in metro/tools/end2end_inference_bodymesh.py and metro/tools/run_metro_bodymesh.py. Is that not the optimal way?

Other than that I am taking tight crops around the ground truth 2d keypoints (with a 1.2x margin) and normalizing.

Hi! Thank you for releasing the code!

When I tried to download datasets with AzCopy, I got this error:

./azcopy copy 'https://datarelease.blob.core.windows.net/metro/datasets/datasets_release/freihand' ~/Downloads --recursive  
INFO: Scanning...

failed to perform copy command due to error: Login Credentials missing. No SAS token or OAuth token is present and the resource is not public

I guess the datasets are not publicly available yet?

All the best,

Kailin

Hello!
Thanks for the great work :)

I have two questions on the loss details in run_metro_bodymesh.py.

1. According to the paper, L1 loss is used to minimize the error between the predicted joints and ground-truth joints. But, torch.nn.MSELoss() is used in run_metro_bodymesh.py instead of L1 loss. Why do you use MSE loss instead of L1 loss described in the paper?
   

2. Why do you subtract the location of pelvis twice? During training, line 207-211 in run_metro_bodymesh.py computes the subtraction of the pelvis location from the given annotation
   
   , but line 125-128 in run_metro_bodymesh.py also computes the subtraction of the pelvis location from the modified annotation obtained by the computation in line 207-211.
   

Could you please check the above questions?
Thanks!

thank for your interesting works. I have a concern about how did you generate the .tsv file for 3DPW dataset? I mean from original dataset. thanks

failed to perform copy command due to error: Login Credentials missing. No SAS token or OAuth token is present and the resource is not publi

MeshTransformer-main/metro/tools/end2end_inference_bodymesh.py --resume_checkpoint ./models/metro_release/metro_3dpw_state_dict.bin --image_file_or_path ./samples/human-body
2021-12-22 20:25:42,376 METRO Inference INFO: Using 1 GPUs
anaconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
anaconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
anaconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
Model name 'metro/modeling/bert/bert-base-uncased/' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc). We assumed 'metro/modeling/bert/bert-base-uncased/' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
File "MeshTransformer-main/metro/tools/end2end_inference_bodymesh.py", line 316, in
main(args)
File "MeshTransformer-main/metro/tools/end2end_inference_bodymesh.py", line 222, in main
config.output_attentions = False
AttributeError: 'NoneType' object has no attribute 'output_attentions'
2021-12-22 20:25:47,352 METRO Inference INFO: Inference: Loading from checkpoint ./models/metro_release/metro_3dpw_state_dict.bin

Hi! Thanks for your great work.
I have some questions about training on 3DPW.

1. According to docs/EXP.md, when train on 3DPW, 3dpw/test_has_gender.yaml is used to evaluate during training:

   --val_yaml 3dpw/test_has_gender.yaml \

   When evaluate on 3DPW after training, 3dpw/test.yaml is used instead:

   --val_yaml 3dpw/test.yaml \

   I only find test_has_gender.yaml in the provided 3dpw.tar archive. Are the above 2 files the same? If not, what are the differences?

2. According to metro/tools/tsv_demo_3dpw.py#L71 and metro/tools/run_metro_bodymesh.py#L395, neutral SMPL model is used to generate GT kp3d and GT vertices, for both 3DPW train set and testing set. I wonder if it is correct because 3DPW provides gender attributes, and only use the model corresponding to the given gender gives the correct output, as far as I know.

Thanks again and looking forward to your reply!

can you please add a colab for inference?

Hi authors,

I found that the original 3DPW dataset has:

• 3D joints with the shape of 24x3
• 2D joints with the shape of 18x3

However, your customized 3DPW dataset has:

• 3D joints with the shape of 24x4
• 2D joints with the shape of 24x3

Therefore, my question is "what is the fourth element of your customized 3D joints"?
Thank you!

What's speed of Hand MeshTransformer compared with other methods such as CNN？

Hi, I'm trying to download the datasets and annotations as described in step #5 here https://github.com/microsoft/MeshTransformer/blob/main/docs/DOWNLOAD.md

This is also referenced by the Mesh Graphormer project.

I'm getting the following error.

failed to perform copy command due to error: Login Credentials missing. No SAS token or OAuth token is present and the resource is not public

Can you please help?
Thanks!

I ran the commond:
python setup.py install --cuda_ext --cpp_ext
and got

torch.__version__  = 1.4.0


setup.py:107: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
  warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")

Compiling cuda extensions with
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
from /usr/local/cuda-10.1/bin

running install
running bdist_egg
running egg_info
writing apex.egg-info/PKG-INFO
writing dependency_links to apex.egg-info/dependency_links.txt
writing top-level names to apex.egg-info/top_level.txt
reading manifest file 'apex.egg-info/SOURCES.txt'
adding license file 'LICENSE'
writing manifest file 'apex.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
running build_ext
building 'fused_layer_norm_cuda' extension
gcc -pthread -B /home/bob/anaconda3/envs/metro/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include/TH -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-10.1/include -I/home/bob/anaconda3/envs/metro/include/python3.7m -c csrc/layer_norm_cuda.cpp -o build/temp.linux-x86_64-3.7/csrc/layer_norm_cuda.o -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_layer_norm_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/usr/local/cuda-10.1/bin/nvcc -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include/TH -I/home/bob/anaconda3/envs/metro/lib/python3.7/site-packages/torch/include/THC -I/usr/local/cuda-10.1/include -I/home/bob/anaconda3/envs/metro/include/python3.7m -c csrc/layer_norm_cuda_kernel.cu -o build/temp.linux-x86_64-3.7/csrc/layer_norm_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -maxrregcount=50 -O3 --use_fast_math -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=fused_layer_norm_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_61,code=sm_61 -std=c++11
csrc/layer_norm_cuda_kernel.cu:4:10: fatal error: ATen/cuda/DeviceUtils.cuh: No such file or directory
 #include "ATen/cuda/DeviceUtils.cuh"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/usr/local/cuda-10.1/bin/nvcc' failed with exit status 1

I have installed CUDA10.1, ran commond:
which nvcc
and I got
/usr/local/cuda-10.1/bin/nvcc

hello, thx for your nice work. I want to know how do you set position embedding? Do you just follow the setting of BERT?

Hi,

First of all, thank you so much for your contribution!

I am facing a problem when running your quick demo code. I hope 3D mesh is generated as shown in your demo.

However, 3D mesh was not generated in my example.

Could you please give me some advice?

Demo for human-body works though, just the hand inference does not seem to work.

021-05-28 14:43:06,634 METRO Inference INFO: Using 1 GPUs
/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/manopth/manolayer.py:72: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
torch.Tensor(smpl_data['betas'].r).unsqueeze(0))
/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/scipy/sparse/_index.py:84: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
2021-05-28 14:43:08,726 METRO Inference INFO: Inference: Loading from checkpoint ./models/metro_release/metro_hand_state_dict.bin
2021-05-28 14:43:08,726 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2021-05-28 14:43:08,726 METRO Inference INFO: Update config parameter hidden_size: 768 -> 1024
2021-05-28 14:43:08,726 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2021-05-28 14:43:08,726 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 4096
2021-05-28 14:43:10,284 METRO Inference INFO: Init model from scratch.
2021-05-28 14:43:10,285 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2021-05-28 14:43:10,285 METRO Inference INFO: Update config parameter hidden_size: 768 -> 256
2021-05-28 14:43:10,285 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2021-05-28 14:43:10,285 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 1024
2021-05-28 14:43:10,508 METRO Inference INFO: Init model from scratch.
2021-05-28 14:43:10,508 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2021-05-28 14:43:10,508 METRO Inference INFO: Update config parameter hidden_size: 768 -> 64
2021-05-28 14:43:10,508 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2021-05-28 14:43:10,508 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 256
2021-05-28 14:43:10,556 METRO Inference INFO: Init model from scratch.
=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth
2021-05-28 14:43:13,671 METRO Inference INFO: => loading hrnet-v2-w64 model
2021-05-28 14:43:13,673 METRO Inference INFO: Transformers total parameters: 101182022
2021-05-28 14:43:13,679 METRO Inference INFO: Backbone total parameters: 128059944
2021-05-28 14:43:13,680 METRO Inference INFO: Loading state dict from checkpoint ./models/metro_release/metro_hand_state_dict.bin
Traceback (most recent call last):
File "./metro/tools/end2end_inference_handmesh.py", line 312, in
main(args)
File "./metro/tools/end2end_inference_handmesh.py", line 279, in main
_metro_network.load_state_dict(state_dict, strict=False)
File "/home/wd-nebojsa-b/miniconda3/envs/metro/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for METRO_Hand_Network:
size mismatch for trans_encoder.0.bert.encoder.layer.0.intermediate.dense.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
size mismatch for trans_encoder.0.bert.encoder.layer.0.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.0.output.dense.weight: copying a param with shape torch.Size([1024, 3072]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.1.intermediate.dense.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
size mismatch for trans_encoder.0.bert.encoder.layer.1.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.1.output.dense.weight: copying a param with shape torch.Size([1024, 3072]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.2.intermediate.dense.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
size mismatch for trans_encoder.0.bert.encoder.layer.2.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.2.output.dense.weight: copying a param with shape torch.Size([1024, 3072]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.3.intermediate.dense.weight: copying a param with shape torch.Size([3072, 1024]) from checkpoint, the shape in current model is torch.Size([4096, 1024]).
size mismatch for trans_encoder.0.bert.encoder.layer.3.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([4096]).
size mismatch for trans_encoder.0.bert.encoder.layer.3.output.dense.weight: copying a param with shape torch.Size([1024, 3072]) from checkpoint, the shape in current model is torch.Size([1024, 4096]).
size mismatch for trans_encoder.1.bert.encoder.layer.0.intermediate.dense.weight: copying a param with shape torch.Size([3072, 256]) from checkpoint, the shape in current model is torch.Size([1024, 256]).
size mismatch for trans_encoder.1.bert.encoder.layer.0.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.0.output.dense.weight: copying a param with shape torch.Size([256, 3072]) from checkpoint, the shape in current model is torch.Size([256, 1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.1.intermediate.dense.weight: copying a param with shape torch.Size([3072, 256]) from checkpoint, the shape in current model is torch.Size([1024, 256]).
size mismatch for trans_encoder.1.bert.encoder.layer.1.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.1.output.dense.weight: copying a param with shape torch.Size([256, 3072]) from checkpoint, the shape in current model is torch.Size([256, 1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.2.intermediate.dense.weight: copying a param with shape torch.Size([3072, 256]) from checkpoint, the shape in current model is torch.Size([1024, 256]).
size mismatch for trans_encoder.1.bert.encoder.layer.2.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.2.output.dense.weight: copying a param with shape torch.Size([256, 3072]) from checkpoint, the shape in current model is torch.Size([256, 1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.3.intermediate.dense.weight: copying a param with shape torch.Size([3072, 256]) from checkpoint, the shape in current model is torch.Size([1024, 256]).
size mismatch for trans_encoder.1.bert.encoder.layer.3.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([1024]).
size mismatch for trans_encoder.1.bert.encoder.layer.3.output.dense.weight: copying a param with shape torch.Size([256, 3072]) from checkpoint, the shape in current model is torch.Size([256, 1024]).
size mismatch for trans_encoder.2.bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([30522, 128]) from checkpoint, the shape in current model is torch.Size([30522, 64]).
size mismatch for trans_encoder.2.bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 128]) from checkpoint, the shape in current model is torch.Size([512, 64]).
size mismatch for trans_encoder.2.bert.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([2, 128]) from checkpoint, the shape in current model is torch.Size([2, 64]).
size mismatch for trans_encoder.2.bert.embeddings.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.embeddings.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.self.query.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.self.query.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.self.key.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.self.key.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.self.value.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.self.value.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.output.dense.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.attention.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.intermediate.dense.weight: copying a param with shape torch.Size([3072, 128]) from checkpoint, the shape in current model is torch.Size([256, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([256]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.output.dense.weight: copying a param with shape torch.Size([128, 3072]) from checkpoint, the shape in current model is torch.Size([64, 256]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.0.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.self.query.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.self.query.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.self.key.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.self.key.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.self.value.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.self.value.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.output.dense.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.attention.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.intermediate.dense.weight: copying a param with shape torch.Size([3072, 128]) from checkpoint, the shape in current model is torch.Size([256, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([256]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.output.dense.weight: copying a param with shape torch.Size([128, 3072]) from checkpoint, the shape in current model is torch.Size([64, 256]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.1.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.self.query.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.self.query.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.self.key.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.self.key.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.self.value.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.self.value.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.output.dense.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.attention.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.intermediate.dense.weight: copying a param with shape torch.Size([3072, 128]) from checkpoint, the shape in current model is torch.Size([256, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([256]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.output.dense.weight: copying a param with shape torch.Size([128, 3072]) from checkpoint, the shape in current model is torch.Size([64, 256]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.2.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.self.query.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.self.query.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.self.key.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.self.key.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.self.value.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.self.value.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.output.dense.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.attention.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.intermediate.dense.weight: copying a param with shape torch.Size([3072, 128]) from checkpoint, the shape in current model is torch.Size([256, 64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.intermediate.dense.bias: copying a param with shape torch.Size([3072]) from checkpoint, the shape in current model is torch.Size([256]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.output.dense.weight: copying a param with shape torch.Size([128, 3072]) from checkpoint, the shape in current model is torch.Size([64, 256]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.output.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.output.LayerNorm.weight: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.encoder.layer.3.output.LayerNorm.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.pooler.dense.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 64]).
size mismatch for trans_encoder.2.bert.pooler.dense.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.bert.position_embeddings.weight: copying a param with shape torch.Size([512, 128]) from checkpoint, the shape in current model is torch.Size([512, 64]).
size mismatch for trans_encoder.2.bert.img_embedding.weight: copying a param with shape torch.Size([128, 128]) from checkpoint, the shape in current model is torch.Size([64, 128]).
size mismatch for trans_encoder.2.bert.img_embedding.bias: copying a param with shape torch.Size([128]) from checkpoint, the shape in current model is torch.Size([64]).
size mismatch for trans_encoder.2.cls_head.weight: copying a param with shape torch.Size([3, 128]) from checkpoint, the shape in current model is torch.Size([3, 64]).
size mismatch for upsampling.weight: copying a param with shape torch.Size([1723, 431]) from checkpoint, the shape in current model is torch.Size([778, 195]).
size mismatch for upsampling.bias: copying a param with shape torch.Size([1723]) from checkpoint, the shape in current model is torch.Size([778]).
size mismatch for cam_param_fc2.weight: copying a param with shape torch.Size([250, 431]) from checkpoint, the shape in current model is torch.Size([150, 216]).
size mismatch for cam_param_fc2.bias: copying a param with shape torch.Size([250]) from checkpoint, the shape in current model is torch.Size([150]).
size mismatch for cam_param_fc3.weight: copying a param with shape torch.Size([3, 250]) from checkpoint, the shape in current model is torch.Size([3, 150]).

Hi!

Once again, kudos for the amazing work, I can't wait for the Mesh Graphormer to get out (hopefully it will).

Is there a part of the code where we can extract 3D Pose (and also maybe betas) from vertices, similarly like you did for 3D Joints (get_joints)?

Cheers!

Hello,

Thank you for the great work.
I run this code and save model up to 10 epochs. And i want to stop learning and restart from 5 epoch.
but 'iteration' restart from 0, not the last iteration. How can i restart learning?

Did you train your model with mixed precision?
I have not seen the code for mixed-precision training.

I have confusion regarding the use of SMPL and MANO in your model. In your paper, you mention that the image feature is concatenated with the 3D coordinates of everybody joint i, and 3D coordinates of every vertex j. Based on the code my understanding is that SMPL and MANO are used for this purpose. But these are themselves parametric models which provide body shape and pose. So why are you concatenating the body mesh and joint vertices obtained from SMPL and MANO to create the body mesh again?

I would really appreciate it if you could further clarify this.