Dear Authors, great work and the results seem to be impressive. Unfortunately I can't get the videopose.py with your provided demo video running as I am facing following error:

the video is 25.051 f/s
Loading YOLO model..
Loading pose model from joints_detectors/Alphapose/models/sppe/duc_se.pth
Start pose estimation...
  0%|                                                                                                                       | 0/338 [00:00<?, ?it/s]Exception in thread Thread-4:
Traceback (most recent call last):
  File "/home/sebastian/anaconda3/envs/video_to_pose3D/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/home/sebastian/anaconda3/envs/video_to_pose3D/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/sebastian/video-to-pose3D/joints_detectors/Alphapose/dataloader.py", line 678, in update
    img = vis_frame(orig_img, result)
  File "/home/sebastian/video-to-pose3D/joints_detectors/Alphapose/fn.py", line 219, in vis_frame
    polygon = cv2.ellipse2Poly((int(mX), int(mY)), (int(length / 2), stickwidth), int(angle), 0, 360, 1)
TypeError: only integer tensors of a single element can be converted to an index
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 338/338 [00:22<00:00, 15.10it/s]
===========================> Rendering remaining images in the queue...
===========================> If this step takes too long, you can enable the --vis_fast flag to use fast rendering (real-time).

I waited for long time but this step does not proceed (the --vis_fast flag does not seem to exist anymore). I am not sure if its a threading or visualization issue - do you have any idea how to solve this?

Thanks and best, Sebastian

Hi
I was wondering if there is a script , that parses out the 3d joints per frame , that can be used for animation purposes or perhaps there is an already exisitng module that does that ?
/Thanks
N

hi:
Sorry to bother. It is a interesting work, but when i run the file named origin_run.py, it has a error about having not 'data/data_3d_h36m.npz'. So doesn't this file exist in your code? How can i geit it? thanks for yourr work and any reply!@zh-plus

running in vscode with a conda enviornment that is setup with all dependencies. I have followed instructions as mentioned in readme.md. Downloaded all weights and requirements. I changed the directory for video file and have changed nothing else. can someone help me with this? I am not training anything. just planning to get 3D pose out of a video (so testing)

So this is the error i am getting:

self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for FastPose:
size mismatch for conv_out.weight: copying a param with shape torch.Size([33, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([17, 128, 3, 3]).
size mismatch for conv_out.bias: copying a param with shape torch.Size([33]) from checkpoint, the shape in current model
is torch.Size([17]).

(pose) D:\PYTHON\pose_detection>python videopose.py
outputs/dance.mp4 --- elapsed time: 3.8129527000000003 s
Traceback (most recent call last):
File "videopose.py", line 164, in
inference_video('outputs/dance.mp4', 'alpha_pose')
File "videopose.py", line 160, in inference_video
main(args)
File "videopose.py", line 69, in main
keypoints = detector_2d(video_name)
File "D:\PYTHON\pose_detection\joints_detectors\Alphapose\gene_npz.py", line 67, in handle_video
data_loader = ImageLoader(im_names, batchSize=args.detbatch, format='yolo').start()
UnboundLocalError: local variable 'im_names' referenced before assignment

Do you have any idea why this is happens ?

请问是否可以在运行过程中实时获取到3D关节的数据？以及是否可以将关节信息导出至外部文件供其他应用使用？

hello! I run your code and got a bug:
MovieWriter stderr: ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory

how to solve it? which version of ffmpeg do you use?
Thank you very much!!!

Here is The Error:
C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose>"C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\python.exe" videopose.py
kunkun_cut.mp4 --- elapsed time: 0.6922928 s
Traceback (most recent call last):
File "videopose.py", line 332, in
inference_video('kunkun_cut.mp4', 'alpha_pose')
File "videopose.py", line 195, in inference_video
main(args)
File "videopose.py", line 67, in main
detector_2d = get_detector_2d(args.detector_2d)
File "videopose.py", line 55, in get_detector_2d
return detector_mapdetector_name
File "videopose.py", line 40, in get_alpha_pose
from joints_detectors.Alphapose.gene_npz import generate_kpts as alpha_pose
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose\joints_detectors\Alphapose\gene_npz.py", line 22, in
self.device = device
NameError: name 'device' is not defined

C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose>command.bat

C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose>"C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\python.exe" videopose.py
the video is 25.051 f/s
Loading YOLO model..
kunkun_cut.mp4 --- elapsed time: 2.4925298000000002 s
Traceback (most recent call last):
File "videopose.py", line 332, in
inference_video('kunkun_cut.mp4', 'alpha_pose')
File "videopose.py", line 195, in inference_video
main(args)
File "videopose.py", line 74, in main
keypoints = detector_2d(video_name)
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose\joints_detectors\Alphapose\gene_npz.py", line 38, in generate_kpts
final_result, video_name = handle_video(video_file)
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose\joints_detectors\Alphapose\gene_npz.py", line 124, in handle_video
det_loader = DetectionLoader(data_loader, batchSize=args.detbatch).start()
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\vidpose\joints_detectors\Alphapose\dataloader.py", line 280, in init
self.det_model.cuda()
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\nn\modules\module.py", line 305, in cuda
return self._apply(lambda t: t.cuda(device))
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\nn\modules\module.py", line 202, in _apply
module._apply(fn)
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\nn\modules\module.py", line 202, in _apply
module._apply(fn)
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\nn\modules\module.py", line 202, in _apply
module._apply(fn)
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\nn\modules\module.py", line 224, in _apply
param_applied = fn(param)
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\nn\modules\module.py", line 305, in
return self.apply(lambda t: t.cuda(device))
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\cuda_init.py", line 192, in _lazy_init
check_driver()
File "C:\Users\Muhammad Wajeeh\Desktop\VoiceCloneApp\lib\site-packages\torch\cuda_init.py", line 102, in _check_driver
http://www.nvidia.com/Download/index.aspx""")
AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx

Can We Run It ON CPU?

I tried to run the code with hr_pose in the following way in the videopose.py file
inference_video('outputs/dance.mp4', 'hr_pose')
Error message

outputs/dance.mp4 --- elapsed time: 3.2093143170004623 s
Traceback (most recent call last):
  File "videopose.py", line 164, in <module>
    inference_video('outputs/dance.mp4', 'hr_pose')
  File "videopose.py", line 160, in inference_video
    main(args)
  File "videopose.py", line 79, in main
    keypoints = normalize_screen_coordinates(keypoints[..., :2], w=1000, h=1002)
  File "/home/viz/test/visualization/video-to-pose3D/common/camera.py", line 16, in normalize_screen_coordinates
    assert X.shape[-1] == 2
AssertionError

这个项目可以预见的帮助小成本动画制作公司低成本进行动作捕捉。希望不要因为律师函而搞砸。
fork为敬。

我试videopose3d的时候用human3.6m出现下面情况
python run.py -k cpn_ft_h36m_dbb -arc 3,3,3,3,3 -c checkpoint --evaluate pretrained_h36m_cpn.bin
Namespace(actions='*', architecture='3,3,3,3,3', batch_size=1024, bone_length_term=True, by_subject=False, causal=False, channels=1024, checkpoint='checkpoint', checkpoint_frequency=10, data_augmentation=True, dataset='h36m', dense=False, disable_optimizations=False, downsample=1, dropout=0.25, epochs=60, evaluate='pretrained_h36m_cpn.bin', export_training_curves=False, keypoints='cpn_ft_h36m_dbb', learning_rate=0.001, linear_projection=False, lr_decay=0.95, no_eval=False, no_proj=False, render=False, resume='', stride=1, subjects_test='S9,S11', subjects_train='S1,S5,S6,S7,S8', subjects_unlabeled='', subset=1, test_time_augmentation=True, viz_action=None, viz_bitrate=3000, viz_camera=0, viz_downsample=1, viz_export=None, viz_limit=-1, viz_no_ground_truth=False, viz_output=None, viz_size=5, viz_skip=0, viz_subject=None, viz_video=None, warmup=1)
Loading dataset...
Preparing data...
Loading 2D detections...
INFO: Receptive field: 243 frames
INFO: Trainable parameter count: 16952371
Loading checkpoint checkpoint\pretrained_h36m_cpn.bin
This model was trained for 80 epochs
INFO: Testing on 543344 frames
Evaluating...
----Directions----
Test time augmentation: True
Protocol #1 Error (MPJPE): 45.156179363731496 mm
Protocol #2 Error (P-MPJPE): 34.119459297758915 mm
Protocol #3 Error (N-MPJPE): 42.62937948308446 mm
Velocity Error (MPJVE): 2.9753524038954198 mm
系统window10

Hello,

I'm working on an action recognition project. Our model based on 25 body joint configurations. I need to use "video-to-pose3D" to generate 3d pose points. I found that "video-to-pose3d" using 17 body joints, not 25.

My question is, Is there is a way to use the 25 body joints? If not, Is there a converter that converts 17 to 25 body joints?

Regards,

When trying to run the script, it eventually arrives at a point where it shows the following message:

===========================> Rendering remaining images in the queue...
===========================> If this step takes too long, you can enable the --vis_fast flag to use fast rendering (real-time).

However, when I try to run the script like this:

python videopose.py --vis_fast

or like this:

python videopose.py --viz_fast

I get the message

videopose.py: error: unrecognized arguments: --vis_fast

or

videopose.py: error: unrecognized arguments: --viz_fast

What am I doing wrong?

Hi researchers:
I want to use Openpose as the 2D Joint detector. However, Openpose generates MISSING_VALUE (0 or -1) when the parts are not shown, which leads to bad results. How can I deal with the problem?

File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
self.run()
File "/home/ubuntu/anaconda3/lib/python3.7/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/video_to_3dmodel/joints_detectors/Alphapose/dataloader.py", line 678, in update
img = vis_frame(orig_img, result)
File "/home/ubuntu/video_to_3dmodel/joints_detectors/Alphapose/fn.py", line 198, in vis_frame
img = cv2.addWeighted(bg, transparency, img, 1 - transparency, 0)
TypeError: Argument 'alpha' can not be treated as a double

Hi , could you also see the possibility to add 2d detectors such as

posenet
and
simpleJHrNet

I was also looking into lifring it from the following repositoy but i had not managed to get good resutls

3d pose this takes open pose as input but i think it could be extended to other detectors as well . What are your thoughts ?

when I try to specify the architecture, it occurs such error:
videopose.py: error: unrecognized arguments: -arc 3,3,3.
How to address it, please?

As i can see there are some demos with Moving Grid in 3D space,
(Input is my Custom video)

When i tried that with my custom Video, rendering is located in centre only.
i am getting this with my input-:
https://raw.githubusercontent.com/facebookresearch/VideoPose3D/master/images/demo_yt.gif

Is it possible to do rendering like Moving-Grid in 3D space on custom video??

I want to get this kind of output with input-:
https://raw.githubusercontent.com/facebookresearch/VideoPose3D/master/images/demo_temporal.gif

can we get that 3D rendering with moving Grid????
if Yes, then what is the approach ???

ref-:facebookresearch/VideoPose3D#147
Thanks

Hi @zh-plus, i am happy to help you with finalizing the openpose detector. As I am using Anaconda I have problems with installing openpose as Anaconda includes a Protobuf version that is incompatible with Caffe. Therefore I thought it would be easier to use this tf openpose implementation. Finally I could write a conda install file to easily set up the system. What do you think?

hello! Which version of cuda and cudnn is installed in your computer environment?

my version is cuda 10.2 and cudnn 7.6.5.

I got a bug: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Could you explain and point to me in your code where you are perform normalization and unNormalization of the data.

If I just try to plot the 3D output the values are all less than 1 just trying to understand where you perform unNormalization and also what sort of preprocessing you apply to the two dimensional keypoints.

Traceback (most recent call last):
File "videopose.py", line 164, in
inference_video('outputs/gait.mp4', 'alpha_pose')
File "videopose.py", line 160, in inference_video
main(args)
File "videopose.py", line 69, in main
keypoints = detector_2d(video_name)
File "/home/makang/code/Pose/video-to-pose3D/joints_detectors/Alphapose/gene_npz.py", line 63, in generate_kpts
kpts = np.array(kpts).astype(np.float32)
ValueError: setting an array element with a sequence.

弱弱地问一句 代码框架是什么 基于alphapose么

hello! how to fix this error?
Thank you!!!

其中 3D Joint detectors - pretrained_h36m_detectron_coco.bin 来自 Facebook Research, 麻烦写清楚

hello！Could you upload the configuration of your environment?
if you use anaconda, you can use this command:
conda activate yourenv
conda list --export > package-list.txt
Thank you very much!!!!
(I encounter a problem that ask me to downgrade my python version to 3.6~3.7, because matplotlib=3.1.0 is incompatible with python3.9)

goole drive打不开，可以提供一下pose_hrnet*的百度网盘链接分享吗。

Hi,
I want to know if there's a parameter(in degree) in videopose.py to change the 3D result viewpoint from default view?
thanks

Hello
I have a few following questions, while playing around with your implementation.
Could you tell ,where the 2D keypoints are stored and the data type for the 2D detector hrnet (which variable exactly).
Second is where this 2d keypoints acts as input for the 3d module ??
Third , is it possible to export the 3d joints with the x,y,z co-ordinate to a csv fileper frame for analysis with pandas for example

使用videopose3D论文提供的模型参数得到的可视化结果很差，pretrained_h36m_detectron_coco.bin使用的是什么数据集？

When I try to run python videopose.py with the demo video, I get the message "Argument 'alpha' can not be treated as a double:

(base) xxx@yyy:~/miniconda3/lib/python3.7/site-packages/video-to-pose3D$ python videopose.py 
the video is 25.051 f/s
Loading YOLO model..
/opt/conda/conda-bld/pytorch_1591914880026/work/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of nonzero is deprecated:
	nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
	nonzero(Tensor input, *, bool as_tuple)
Loading pose model from joints_detectors/Alphapose/models/sppe/duc_se.pth
Start pose estimation...
  0%|▎                                                                                                    | 1/338 [00:00<00:46,  7.18it/s]Exception in thread Thread-4:
Traceback (most recent call last):
  File "/home/xxx/miniconda3/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/xxx/miniconda3/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/xxx/miniconda3/lib/python3.7/site-packages/video-to-pose3D/joints_detectors/Alphapose/dataloader.py", line 678, in update
    img = vis_frame(orig_img, result)
  File "/home/xxx/miniconda3/lib/python3.7/site-packages/video-to-pose3D/joints_detectors/Alphapose/fn.py", line 198, in vis_frame
    img = cv2.addWeighted(bg, transparency, img, 1 - transparency, 0)
TypeError: Argument 'alpha' can not be treated as a double

The script seems to continue, though I'm not sure if it eventually finishes, since I had to interrupt.

how do you train your model (retrained_h36m_detectron_coco.bin). I use the official model can not achieve your performance of visualizasion

save 3D joint points

np.save('outputs/test_3d_output.npy', prediction, allow_pickle=True)
rot = np.array([0.14070565, -0.15007018, -0.7552408, 0.62232804], dtype=np.float32)
prediction = camera_to_world(prediction, R=rot, t=0)

请问，这部分代码中，rot这四个参数代表什么呢，如何可以得到它们?

Hi I was trying to run videopose.py using python 3.7 in virtual env using ubuntu 18.04. The command i used was as follows

python3.7 videopose.py --video ./outputs/kunkun_cut.mp4 which results in the following error

Traceback (most recent call last):
  File "videopose.py", line 174, in <module>
    inference_video('outputs/dance.mp4', 'alpha_pose')
  File "videopose.py", line 163, in inference_video
    main(args)
  File "videopose.py", line 67, in main
    split_video(args.viz_video)
  File "/sandbox/video-to-pose3D/common/utils.py", line 154, in split_video
    saved_path = os.path.dirname(save_path)
UnboundLocalError: local variable 'save_path' referenced before assignment

I am using hrnet as 2D detector .

Hi there~
Thank you for sharing such great work to us.
It is really helpful for many people.

When I run $ python videopose.py below
It will show an error message which is (AttributeError: 'list' object has no attribute 'shape')
Could you please help me to find out the error
Thank you very much!

=================================================================================================
the video is 25.051 f/s
Loading YOLO model..
/home/xxx/.conda/envs/3d/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:129: UserWarning: nn.Upsample is deprecated. Use nn.functional.interpolate instead.
warnings.warn("nn.{} is deprecated. Use nn.functional.interpolate instead.".format(self.name))
Loading pose model from joints_detectors/Alphapose/models/sppe/duc_se.pth
Start pose estimation...
100%|█████████████████████████████████████████| 338/338 [00:23<00:00, 14.11it/s]
kpts npz save in outputs/alpha_pose_kunkun_cut/kunkun_cut.npz
-------------- load data spends 27.51 seconds
Loading checkpoint checkpoint/pretrained_h36m_detectron_coco.bin
-------------- load 3D model spends 0.02 seconds
Rendering...
-------------- generate reconstruction 3D data spends 0.15 seconds
===========================> This video get 338 frames in total.
0%| | 0/338 [00:00<?, ?it/s]/home/xxx/video-to-pose3D/common/visualization.py:181: UserWarning: Tight layout not applied. The left and right margins cannot be made large enough to accommodate all axes decorations.
fig.tight_layout()
0%|▏ | 1/338 [00:00<00:43, 7.82it/s]outputs/kunkun_cut.mp4 --- elapsed time: 28.11254884302616 s
Traceback (most recent call last):
File "videopose.py", line 164, in
inference_video('outputs/kunkun_cut.mp4', 'alpha_pose')
File "videopose.py", line 160, in inference_video
main(args)
File "videopose.py", line 133, in main
input_video_skip=args.viz_skip)
File "/home/xxx/video-to-pose3D/common/visualization.py", line 187, in render_animation
anim.save(output, writer=writer)
File "/home/xxx/.conda/envs/3d/lib/python3.6/site-packages/matplotlib/animation.py", line 1133, in save
anim._draw_next_frame(d, blit=False)
File "/home/xxx/.conda/envs/3d/lib/python3.6/site-packages/matplotlib/animation.py", line 1168, in _draw_next_frame
self._draw_frame(framedata)
File "/home/xxx/.conda/envs/3d/lib/python3.6/site-packages/matplotlib/animation.py", line 1718, in _draw_frame
self._drawn_artists = self._func(framedata, *self._args)
File "/home/xxx/video-to-pose3D/common/visualization.py", line 175, in update_video
lines_3d[n][j - 1][0].set_3d_properties([pos[j, 2], pos[j_parent, 2]], zdir='z')
File "/home/xxx/.conda/envs/3d/lib/python3.6/site-packages/mpl_toolkits/mplot3d/art3d.py", line 143, in set_3d_properties
zs = np.broadcast_to(zs, xs.shape)
AttributeError: 'list' object has no attribute 'shape'
0%|▏ | 1/338 [00:00<00:56, 5.99it/s]

Hi @zh-plus, I am wondering where the mapping from the 17 Coco keypoints generated by AlphaPose to the HM3.6 keypoints for inferring the 3D uplifting takes place? It's easy to determine HM3.6 keypoints PELVIS and THORAX from Cocos keypoints (RHip + LHip) /2 and (RShoulder + LShoulder) /2 respectively, but how are the HM3.6 SPINE and HEAD keypoints determined? I could not find the code snippet for mapping keypoints in your repo..

COCO keypoints
{0, "Nose"},
{1, "LEye"},
{2, "REye"},
{3, "LEar"},
{4, "REar"},
{5, "LShoulder"},
{6, "RShoulder"},
{7, "LElbow"},
{8, "RElbow"},
{9, "LWrist"},
{10, "RWrist"},
{11, "LHip"},
{12, "RHip"},
{13, "LKnee"},
{14, "Rknee"},
{15, "LAnkle"},
{16, "RAnkle"}

HM3.6 keypoints
PELVIS = 0
R_HIP = 1
R_KNEE = 2
R_FOOT = 3
L_HIP = 4
L_KNEE = 5
L_FOOT = 6
SPINE = 7
THORAX = 8
NOSE = 9
HEAD = 10
L_SHOULDER = 11
L_ELBOW = 12
L_WRIST = 13
R_SHOULDER = 14
R_ELBOW = 15
R_WRIST = 16

Hi,

I tried the code to run in window. Everything was fine but the main issue i am facing is that the code doesn't utilize the GPU. I tried each and every possible solution but helpless.
Any idea?

律师函警告

json文件里，深度用小数表示，单位是米吗？

I run your code on a server, but I can't get output. Could you please help me find out what problems with it?

`the video is 25.051 f/s
Loading YOLO model..
Loading pose model from joints_detectors/Alphapose/models/sppe/duc_se.pth
Start pose estimation...
0%| | 0/338 [00:00<?, ?it/s]Exception in thread Thread-4:
Traceback (most recent call last):
File "/home/hedian/anaconda3/envs/videopose3d/lib/python3.9/threading.py", line 950, in _bootstrap_inner
self.run()
File "/home/hedian/anaconda3/envs/videopose3d/lib/python3.9/threading.py", line 888, in run
self._target(*self._args, **self._kwargs)
File "/home/hedian/project/video-to-pose3D/joints_detectors/Alphapose/dataloader.py", line 678, in update
img = vis_frame(orig_img, result)
File "/home/hedian/project/video-to-pose3D/joints_detectors/Alphapose/fn.py", line 198, in vis_frame
img = cv2.addWeighted(bg, transparency, img, 1 - transparency, 0)
cv2.error: OpenCV(4.5.3) 👎 error: (-5:Bad argument) in function 'addWeighted'

Overload resolution failed:

• Argument 'alpha' can not be treated as a double
• Argument 'alpha' can not be treated as a double

100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 338/338 [00:20<00:00, 16.19it/s]
===========================> Rendering remaining images in the queue...
===========================> If this step takes too long, you can enable the --vis_fast flag to use fast rendering (real-time).`

Hi @zh-plus, I also recently thought of using the Super-SloMo model to increase fps which hopefully yields in a more stable + accurate 3D skeleton tracking. Do you already have a pipeline in mind? Something like adding a new flag for setting the desired fps -> preprocess video using Super-SloMo -> feed 2D detector -> 3D ? I am happy to help you implementing it :)