openxrlab / xrmocap Goto Github PK

In argument list of function visualize_smpl_data, if smpl_data.get_batch_size() > batch_size, there will be no video output. The reason is absent of images_to_video function, which should be called at here. We cannot wait for XRPrimer release forever, shall we copy images_to_video from XRPrimer to XRMoCap and call it inside XRMoCap with warning?

运行

python tools/mview_mperson_topdown_estimator.py \
      --estimator_config 'configs/mvpose_tracking/mview_mperson_topdown_estimator.py' \
      --image_and_camera_param 'xrmocap_data/Shelf_50/image_and_camera_param.txt' \
      --start_frame 300 \
      --end_frame 350 \
      --output_dir 'output/estimation' \
      --enable_log_file

报错：

Traceback (most recent call last):
  File "tools/mview_mperson_topdown_estimator.py", line 17, in <module>
    from xrmocap.core.estimation.builder import build_estimator
  File "/home/xrmocap/xrmocap/core/estimation/__init__.py", line 3, in <module>
    from .mperson_smpl_estimator import MultiPersonSMPLEstimator
  File "/home/xrmocap/xrmocap/core/estimation/mperson_smpl_estimator.py", line 5, in <module>
    from xrprimer.data_structure import Keypoints
ImportError: cannot import name 'Keypoints' from 'xrprimer.data_structure' (/root/miniconda3/envs/xrmocap/lib/python3.8/site-packages/xrprimer/data_structure/__init__.py)

ImportError: cannot import name 'Keypoints' from 'xrprimer.data_structure'
安装过程完全按照教程，但出现以下问题，请问会是什么问题？谢谢

sh ./scripts/eval_mvp.sh 4 configs/mvp/shelf_config/mvp_shelf.py weight/xrmocap_mvp_shelf.pth.tar

shelf weight downloaded from here, and rename to xrmocap_mvp_shelf.pth.jar

got:
`
2022-09-09 07:54:20,180 - mvp_eval - WARNING - The model and loaded state dict do not match exactly

size mismatch for view_embed: copying a param with shape torch.Size([3, 256]) from checkpoint, the shape in current model is torch.Size([5, 256]).
size mismatch for reference_feats.weight: copying a param with shape torch.Size([256, 2304]) from checkpoint, the shape in current model is torch.Size([256, 3840]).
size mismatch for decoder.layers.0.fuse_view_projction.weight: copying a param with shape torch.Size([256, 768]) from checkpoint, the shape in current model is torch.Size([256, 1280]).
size mismatch for decoder.layers.1.fuse_view_projction.weight: copying a param with shape torch.Size([256, 768]) from checkpoint, the shape in current model is torch.Size([256, 1280]).
size mismatch for decoder.layers.2.fuse_view_projction.weight: copying a param with shape torch.Size([256, 768]) from checkpoint, the shape in current model is torch.Size([256, 1280]).
size mismatch for decoder.layers.3.fuse_view_projction.weight: copying a param with shape torch.Size([256, 768]) from checkpoint, the shape in current model is torch.Size([256, 1280]).
size mismatch for decoder.layers.4.fuse_view_projction.weight: copying a param with shape torch.Size([256, 768]) from checkpoint, the shape in current model is torch.Size([256, 1280]).
size mismatch for decoder.layers.5.fuse_view_projction.weight: copying a param with shape torch.Size([256, 768]) from checkpoint, the shape in current model is torch.Size([256, 1280]).
`

Is the code for the end-to-end SMPL (body mesh recovery) mentioned in the MvP paper available?

excusme, does this code surpport batch_size>1? for example, in position_encoding.py code:
pixel_camera = torch.bmm(xy1.flatten(1, 2).repeat(views, 1, 1), torch.inverse(K).transpose(2, 1))
will get error if batch_size>1, because the dim 0 do not equal. Should i replace ".repeat(views, 1, 1)" to "" .repeat(batch*views, 1, 1)"?

Hi, thanks for your work! How can I generate multi-view multi-person full body keypoints3d? Thank very much.

displacement shape is [batch_size, N_VERTS, 3] is fine at xrmocap/model/registrant/smplifyxd.py#L36.
but define error shape [batch_size, NUM_VERTS] at xrmocap/data_structure/body_model/smplxd_data.py#L55

Thanks for your great work!

I tried your demo on single person estimation and it works well. However, it is using smpl for estimation. I wonder how I can change the model from smpl to smplx. How should I modify the scripts? Thanks!

• Created time: 2023-03-01 01:31:29

• Time base: UTC +0

Count

Labels statistics

Most comments

Thanks for your work ! I want to test the model on my own data. So,how to test the model on the custom image ?

Hi, I want to test xrmocap on CMU Panoptic format data. Is there any api for that? Or can u give me some advice for that?

Hi there,

when using the whole shelf dataset und running

python tools/mview_mperson_end2end_estimator.py --output_dir ./output/estimation --model_dir weight/mvp/xrmocap_mvp_shelf-22d1b5ed_20220831.pth --estimator_config configs/modules/core/estimation/mview_mperson_end2end_estimator.py --image_and_camera_param ./xrmocap_data/Shelf/image_and_camera_param.txt --start_frame 0 --end_frame 200 --enable_log_file

i get the following error:
File "/home/pschuelein/projects/xrmocap/tools/mview_mperson_end2end_estimator.py", line 187, in
main(args)
File "/home/pschuelein/projects/xrmocap/tools/mview_mperson_end2end_estimator.py", line 61, in main
pred_keypoints3d, smpl_data_list = smpl_estimator.run(
File "/home/pschuelein/projects/xrmocap/xrmocap/core/estimation/mview_mperson_end2end_estimator.py", line 286, in run
pred_keypoints3d = self.optimize_keypoints3d(pred_keypoints3d)
File "/home/pschuelein/projects/xrmocap/xrmocap/core/estimation/mperson_smpl_estimator.py", line 79, in optimize_keypoints3d
keypoints3d = optimizer.optimize_keypoints3d(
File "/home/pschuelein/projects/xrmocap/xrmocap/transform/keypoints3d/optim/rm_duplicate.py", line 118, in optimize_keypoints3d
kps3d_optim[frame_idx, frame_identities,
IndexError: index 10 is out of bounds for axis 1 with size 10

I have put the used shelf dataset in the same folder structure as the example Shelf_50 testset.

The error also only occurs when frame 150 is included. Maybe this helps to isolate the error happening here.

Thanks for your help.

Please consider adding Human3.6M support as many paper used it for evaluation.

link: http://vision.imar.ro/human3.6m/description.php

• Created time: 2023-04-01 01:29:40

• Time base: UTC +0

Count

Labels statistics

Most comments

• Created time: 2023-05-01 01:30:19

• Time base: UTC +0

Count

Labels statistics

Most comments

Thanks for your work, Can you provide a tutorial on customer data？

1. LBFGS raises an error when input has only one frame. A keypoints3d of two frames is good.
2. Argument img_paths does not work in mview_sperson_smpl_estimator
3. Error raised if mediapipe has not been installed. https://github.com/openxrlab/xrmocap/blob/main/xrmocap/human_perception/keypoints_estimation/mediapipe_estimator.py#L3

Missing __init__.py at xrmocap/model/registrant/handler.

When I test MvP in Campus dataset. I run the following command:

And I meet the following error:

Can you help me?

Length of mask is 1 when a SMPLData instance is being constructed with default arguments. After it loads SMPLify results(or results from body_model), everything changes except mask. Below is a script to reproduce the issue.

import numpy as np
from xrprimer.utils.log_utils import get_logger
from xrmocap.data_structure.body_model import SMPLData

logger = get_logger()
smpl_data = SMPLData()
logger.info(
    'Init:\n' +
    f'batch_size: {smpl_data.get_batch_size()}\n' +
    f'mask len: {smpl_data.get_mask().shape[0]}\n')

param_dict = dict(
    global_orient=np.zeros((10, 3)),
    transl=np.zeros((10, 3)),
    body_pose=np.zeros((10, 23, 3)),
    betas=np.zeros((10, 10)),
)
smpl_data.from_param_dict(param_dict)
logger.info(
    'After loading param_dict:\n' +
    f'batch_size: {smpl_data.get_batch_size()}\n' +
    f'mask len: {smpl_data.get_mask().shape[0]}\n')

And the output is:

2023-03-02 09:30:04,734 - root - INFO - Init:
batch_size: 1
mask len: 1

2023-03-02 09:30:04,734 - root - INFO - After loading param_dict:
batch_size: 10
mask len: 1

Thanks for your work! How can I visualize the 3d data in a world coordinate system? like this:

好奇，测试结果抖吗？有视频转bvh或者fbx实现？ 想测试一下。

Following the getting_started guidelines, it aims to

• run unit tests within 30 minutes
• complete inference and test tests within 2 hours
• train a basic model within 4 hours

• Created time: 2023-01-01 01:30:54

• Time base: UTC +0

Count

Labels statistics

Most comments

When I use the defalt 2d keypoints detector i.e. mmpose_hrnet_w48_coco_wholebody_384x288_dark_plus.py
I get a wrong 2d keypoint result about the big_toe and small_toe points.
I wonder if it is the kpts detector's fault or some other resons?

Hi,

First of all, thanks for the great work of openxrlab!

I'm testing MVP estimation with default Panoptic 5 views. I'm struggling with inconsistent 3d keypoints estimation.

In the converter configuration, for bbox_detector I tried both "MMtrackDetector", and "MMdetDetector", and for kps2d_estimator I'm using default "MMposeTopDownEstimator" with pretrained "hrnet_w48_coco_wholebody". From the perception 2d view of detected keypoints, it seems that "MMdetDetector" are more sensitive to hidden bodies and produces more small noisy poses.

And the final result of MVP with "MMdetDetector" is as follows. "MMtrackDetector" also has similar inconsistent 3d keypoints.

I'm following xrmocap/configs/modules/core/estimation/mview_mperson_end2end_estimator.py but still got inconsistent 3d keypoints. I guess it might be due to some inappropriate hyperparameters. Would appreciate some suggestions.

Best regards.

Hello! I wonder how to implement online motion capture base on this project? Thanks in advance!

您好，请问本项目支持在线实时动捕么？
通过阅读论文‘Fast and Robust Multi-Person 3D Pose Estimation and Tracking From Multiple Views’，我想尝试修改项目中的mvpose_tracking相关代码，从而实现实时在线动捕，不知道这样做是否可行呢？希望能够得到您的建议，谢谢！

When we have initial betas and the frame size is larger than 1, betas will keep its original shape according to this line, no matter whether use_one_betas_per_video is true or false.

In this way, if initial betas is (21, 10), then the fitted betas has 21 different values even when use_one_betas_per_video=True.
The expected behavior is when use_one_betas_per_video=True, the fitted betas across frames the same, with actual shape (1, 10).

For example,

if self.use_one_betas_per_video:
    if 'betas' not in init_dict:
        betas = torch.zeros(1, self.body_model.betas.shape[-1]).to(
                self.device)
    else:
        betas = init_dict['betas'].mean(dim=0, keepdim=True)
    ret_dict['betas'] = betas

Can you open the tools to make custom mview smc file ?

I want to get the humman dataset. And build the single person

Multiple People demo:
python tools/mview_mperson_topdown_estimator.py \ --estimator_config 'configs/mvpose_tracking/mview_mperson_topdown_estimator.py' \ --image_and_camera_param 'xrmocap_data/Shelf_50/image_and_camera_param.txt' \ --start_frame 300 \ --end_frame 350 \ --output_dir 'output/estimation' \ --enable_log_file
error:
- ERROR - use_shoulder_hip_only is deprecated, please manually set: body_weight=0.0, face_weight=0.0, hand_weight=0.0, shoulder_weight=1.0, hip_weight=1.0 to make sure correct weights are set.

hello, thanks for your excellent project. I wonder whether the multi-view motion capture with human3.6M will be supported? e.g. did you train MvP on Human3.6m dataset?
Thanks for your attention

• Created time: 2022-12-01 01:32:07

• Time base: UTC +0

Count

Labels statistics

Most comments

• Created time: 2023-02-01 01:31:23

• Time base: UTC +0

Count

Labels statistics

Most comments

2023-02-13 16:06:36,471 - __main__ - INFO - Estimating keypoints3d.
2023-02-13 16:06:36,471 - __main__ - ERROR - No one has been detected in any view.
Traceback (most recent call last):
  File "/home/code/xrmocap/tools/process_smc.py", line 214, in <module>
    main(args)
  File "/home/code/xrmocap/tools/process_smc.py", line 76, in main
    keypoints2d_list, keypoints3d, smpl_data = mview_sp_smpl_estimator.run(
  File "/home/code/xrmocap/xrmocap/core/estimation/mview_sperson_smpl_estimator.py", line 232, in run
    keypoints3d = self.estimate_keypoints3d(
  File "/home/code/xrmocap/xrmocap/core/estimation/mview_sperson_smpl_estimator.py", line 330, in estimate_keypoints3d
    raise AttributeError
AttributeError

cmd

python tools/process_smc.py \
	--estimator_config configs/humman_mocap/mview_sperson_smpl_estimator.py \
	--smc_path xrmocap_data/humman/p000127_a000007.smc \
	--output_dir xrmocap_data/humman/p000127_a000007_output \
	--visualize

• Created time: 2023-06-01 01:33:11

• Time base: UTC +0

Count

Labels statistics

Most comments

• Created time: 2022-11-01 03:29:38

• Time base: UTC +0

Count

Labels statistics

Most comments

Could you provide an example of this Python command?

Hi there.

When running the demo after following the install instructions from https://github.com/openxrlab/xrmocap/blob/main/docs/en/installation.md i get the following error. The same error also occurs when running pytest tests/.

Looking forward to your help. Thanks a lot in advance.

python tools/mview_mperson_topdown_estimator.py --estimator_config 'configs/mvpose_tracking/mview_mperson_topdown_estimator.py' --image_and_camera_param 'xrmocap_data/Shelf_50/image_and_camera_param.txt' --start_frame 300 --end_frame 350 --output_dir 'output/estimation' --enable_log_file

`
load checkpoint from local path: weight/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth
Traceback (most recent call last):
File "/home/pschuelein/anaconda3/envs/xrmocap/lib/python3.8/site-packages/mmcv/utils/registry.py", line 69, in build_from_cfg
return obj_cls(**args)
File "/home/pschuelein/projects/xrmocap/xrmocap/human_perception/keypoints_estimation/mmpose_top_down_estimator.py", line 50, in init
self.pose_model = init_pose_model(**mmpose_kwargs)
NameError: name 'init_pose_model' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/pschuelein/anaconda3/envs/xrmocap/lib/python3.8/site-packages/mmcv/utils/registry.py", line 69, in build_from_cfg
return obj_cls(**args)
File "/home/pschuelein/projects/xrmocap/xrmocap/core/estimation/mview_mperson_topdown_estimator.py", line 111, in init
self.kps2d_estimator = build_detector(kps2d_estimator)
File "/home/pschuelein/projects/xrmocap/xrmocap/human_perception/builder.py", line 32, in build_detector
return DETECTORS.build(cfg)
File "/home/pschuelein/anaconda3/envs/xrmocap/lib/python3.8/site-packages/mmcv/utils/registry.py", line 237, in build
return self.build_func(*args, **kwargs, registry=self)
File "/home/pschuelein/anaconda3/envs/xrmocap/lib/python3.8/site-packages/mmcv/utils/registry.py", line 72, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
NameError: MMposeTopDownEstimator: name 'init_pose_model' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "tools/mview_mperson_topdown_estimator.py", line 178, in
main(args)
File "tools/mview_mperson_topdown_estimator.py", line 35, in main
smpl_estimator = build_estimator(estimator_config)
File "/home/pschuelein/projects/xrmocap/xrmocap/core/estimation/builder.py", line 32, in build_estimator
return ESTIMATORS.build(cfg)
File "/home/pschuelein/anaconda3/envs/xrmocap/lib/python3.8/site-packages/mmcv/utils/registry.py", line 237, in build
return self.build_func(*args, **kwargs, registry=self)
File "/home/pschuelein/anaconda3/envs/xrmocap/lib/python3.8/site-packages/mmcv/utils/registry.py", line 72, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
NameError: MultiViewMultiPersonTopDownEstimator: MMposeTopDownEstimator: name 'init_pose_model' is not defined`