yasaminjafarian / hdnet_tiktok Goto Github PK

Hi Yasamin, awesome work! I'm trying to train the model on my own dataset, and I wonder if you could clarify the formats of the labeled data. Specifically, for normal, what should the channel 1,2,3 of the txt files each store? And how did you convert them into visualization?

Dear Yasamin,
In the paper it is mentioned that your model is trained for 380 epoch with batch size 10. What is the number of iterations per epoch there?
Did you train normal estimator and depth estimator separately beforehand?
Did you use renderpeople samples and tiktok pair samples together as in the training code you provided for HDNet training?
I am actually trying to estimate how much time is needed to train the pretrained model with tiktok dataset and tang dataset.

Hi Yasmin,

Amazing work!
If we have SMPL pose parameters for two frames I and j, can we get W_{I->j}^{k} and instead of estimating from least squares solution?

Thanks in advance for any help.

File "/fast/liubl/workspace/HDNet_TikTok/hourglass_net_depth.py", line 119, in hourglass_refinement
out0_d = hourglass_stack_no_incep(netIN)
File "/fast/liubl/workspace/HDNet_TikTok/hourglass_net_depth.py", line 74, in hourglass_stack_no_incep
c0 = conv_layer(stack_in,layer_name('conv'),KER_SZ,NUM_CH[0],bn=bn,training=training)
File "/fast/liubl/workspace/HDNet_TikTok/hourglass_net_depth.py", line 34, in conv_layer
kernel = variable('weights', [kernel_size, kernel_size, input_channels, output_channels], initializer, regularizer=tf.contrib.layers.l2_regularizer(0.0005))
File "/fast/liubl/workspace/HDNet_TikTok/hourglass_net_depth.py", line 28, in variable
return tf.get_variable(name, shape, initializer=initializer, regularizer=regularizer, dtype=tf.float32, trainable=True)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable
aggregation=aggregation)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable
aggregation=aggregation)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable
aggregation=aggregation)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter
aggregation=aggregation)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 929, in _get_single_variable
aggregation=aggregation)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 259, in call
return cls._variable_v1_call(*args, **kwargs)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
shape=shape)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 198, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2511, in default_variable_creator
shape=shape)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 263, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1568, in init
shape=shape)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1698, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 901, in
partition_info=partition_info)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/contrib/layers/python/layers/initializers.py", line 150, in _initializer
seed=seed)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 178, in truncated_normal
shape_tensor, dtype, seed=seed1, seed2=seed2)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 1013, in truncated_normal
name=name)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/fast/liubl/anaconda3/envs/tiktok2/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

Seems not support for current CUDA version

Hello, author, I would like to know how the dataset (Vlasic et al.dataset, THuman2.0 dataset) you are evluating, how to perform ray tracing from mesh to depth map, can you provide relevant code or links or related method implementation details.

Hi! Is it possible to run your code on machine with CUDA 11? I get this error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal: Could not satisfy explicit device specification '' because the node {{colocation_node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0].

and I suspect that it's connected to difference in CUDA version with the version mentioned in README.

Hi, Thank you for sharing this great work!
Do you have any ideas for future work such as fusion multi-view of mesh results, and Bust-Waist-Hip measurements based on these achievements? I'm really interested in this future work. Please notify me if you publish a corresponding paper/repository in this field.

Hi, thanks for the code. I have a question about getting normal map from *_normal_1/2/3.txt.

Based on readme, seems I can simply concatenate the *_normal_1/2/3.txt by the third axis. but I wonder why here, before converting to RGB, you have such conversion (first negative then scale z). Does it mean I have to do the same conversion if I want to use the normal map?

Thank you!

Hi again, we realized that in some of the densepose results, it seems like two dp results are stitched together. I wonder if you perhaps combined all the densepose detections for a single image. Is this a preprocessing mistake or intentional? Because I think it results in detrimental densepose inputs to the model.
We have a script to generate densepose results using detectron2 densepose implementation and best DensePose model with resnet101 bakcbone and DeepLabV3 head. The results are strikingly better as you can see from the below example:

I wonder if the model would improve if we train with these densepose inputs, instead of the ones provided in TikTok dataset.

There was a problem in the process of making an image pair. According to the paper, "At each time instant, we make five image pairs by randomly selecting time instances that have the UV coordinates of at least five common visible body parts while each contains more than 50 overlapping UV coordinates." Does "overlapping UV coordinates" here mean that the exact same number of UV points is more than 50? all just similar points.

Hi Yasamin,

Great to know your wonderful work! Will you release the code for rendering 'RenderPeople dataset' with ' ray tracing algorithm to compute the ground truth depth and render the human textured model'?

Thanks!

Not sure what is going on here. I am running the new NVIDIA 3080 TI. It says my GPU is not supported which I have never had happen to me before even on older codebases. :/
`
Traceback (most recent call last):
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in do_call
return fn(*args)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1339, in run_fn
self.extend_graph()
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1374, in extend_graph
tf_session.ExtendSession(self.session)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal: Could not satisfy explicit device specification '' because the node {{colocation_node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[]
L2Loss: CPU XLA_CPU
Assign: CPU
VariableV2: CPU
Identity: CPU XLA_CPU
Add: CPU XLA_CPU
Mul: CPU XLA_CPU
TruncatedNormal: CPU XLA_CPU
Const: CPU XLA_CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/shape (Const)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mean (Const)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/stddev (Const)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (TruncatedNormal)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mul (Mul)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal (Add)
hourglass_stack_fused_depth_prediction/conv1/weights (VariableV2) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Assign (Assign) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/read (Identity) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/scale (Const) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/L2Loss (L2Loss) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer (Mul) /device:GPU:0

 [[{{node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "HDNet_Inference.py", line 64, in
tf.global_variables_initializer().run()
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2679, in run
run_using_default_session(self, feed_dict, self.graph, session)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 5614, in run_using_default_session
session.run(operation, feed_dict)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in run
feed_dict_tensor, options, run_metadata)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in do_run
run_metadata)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal: Could not satisfy explicit device specification '' because the node node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (defined at /home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:28) placed on device Device assignments active during op 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal' creation:
with tf.device(None): </home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py:1696>
with tf.device(/gpu:0): </home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:26> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0].
Colocation Debug Info:
Colocation group had the following types and supported devices:
Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[]
L2Loss: CPU XLA_CPU
Assign: CPU
VariableV2: CPU
Identity: CPU XLA_CPU
Add: CPU XLA_CPU
Mul: CPU XLA_CPU
TruncatedNormal: CPU XLA_CPU
Const: CPU XLA_CPU

Colocation members, user-requested devices, and framework assigned devices, if any:
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/shape (Const)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mean (Const)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/stddev (Const)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (TruncatedNormal)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mul (Mul)
hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal (Add)
hourglass_stack_fused_depth_prediction/conv1/weights (VariableV2) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Assign (Assign) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/read (Identity) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/scale (Const) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/L2Loss (L2Loss) /device:GPU:0
hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer (Mul) /device:GPU:0

 [[node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (defined at /home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:28) ]]Additional information about colocations:No node-device colocations were active during op 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal' creation.

Device assignments active during op 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal' creation:
with tf.device(None): </home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py:1696>
with tf.device(/gpu:0): </home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:26>

Original stack trace for 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal':
File "HDNet_Inference.py", line 57, in
out2_1 = hourglass_refinement(x1,True)
File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 119, in hourglass_refinement
out0_d = hourglass_stack_no_incep(netIN)
File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 74, in hourglass_stack_no_incep
c0 = conv_layer(stack_in,layer_name('conv'),KER_SZ,NUM_CH[0],bn=bn,training=training)
File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 34, in conv_layer
kernel = variable('weights', [kernel_size, kernel_size, input_channels, output_channels], initializer, regularizer=tf.contrib.layers.l2_regularizer(0.0005))
File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 28, in variable
return tf.get_variable(name, shape, initializer=initializer, regularizer=regularizer, dtype=tf.float32, trainable=True)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable
aggregation=aggregation)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable
aggregation=aggregation)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable
aggregation=aggregation)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter
aggregation=aggregation)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 929, in _get_single_variable
aggregation=aggregation)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 259, in call
return cls._variable_v1_call(*args, **kwargs)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call
shape=shape)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 198, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2511, in default_variable_creator
shape=shape)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 263, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1568, in init
shape=shape)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1698, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 901, in
partition_info=partition_info)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/contrib/layers/python/layers/initializers.py", line 150, in _initializer
seed=seed)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 178, in truncated_normal
shape_tensor, dtype, seed=seed1, seed2=seed2)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 1013, in truncated_normal
name=name)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

`

Hello!
I would like to first appreciate sharing your great work! It is a very impressive and wonderful work! :)

May I ask which method is used for extracting the foreground mask?

Thank you!

The models provided for inference and training seem to be the same. Is the model trained on RenderPeople and TikTok dataset or only RenderPeople?

When I change the variable BATCH_SIZE in training_HDNet.py, it raises an error in Depth2Points3D_transformed_vector. The issue is there is an another batch_size parameter in the line 51 of the same function.
Even if I set it to be the same as BATCH_SIZE, the training wouldn't work as intended because, tiktok batches are discarded in transform_depth_PCs_dp_based2. Since each TikTok sample will have different number of correspondences, I guess the best solution is iterating over batches inside transform_depth_PCs_dp_based2.

It seems that DensePose is required, but it's discreetly mentioned in the inference section. This repository requires DensePose, therefore it should be listed under requirements.

Thanks for sharing.
I downloaded the training data you provide, and use the Eq (4) https://github.com/yasaminjafarian/HDNet_TikTok/blob/main/training/training_code/utils/Geometry_MB.py#L94 to convert a depth map to a normal map.
The results show below:

From left to right are: Image, depth map, gt normal map(N1) and convert normal map(N2).
The error between N1 and N2 is 4120.162. Error equals to (abs(N1-N2).sum()), The pixel value of N1 and N2 range from 0 to 1.
I use a GT depth and covert it to a normal map. I think the difference between N1 and N2 should not be that huge.
I wonder why? Or I misunderstood something?
Looking forward to your reply.

Hi! I'm trying to download the pretrained model either via wget (provided command in the README) or directly on the google drive webpage, both failed.

wget reports:

On the webpage Google Drive says:

Could you help out here? Thanks.

Does the inference demo just use the tang datasets (pre-training model )instead of the HDNet model，So the reasoning effect is not very good？

I'm interested in the new view video generated with the estimated depth, would you please share some methods to do this. Again, your work is really amazing!

Hi yasamin,thanks to your wonderful works.

When I run your Inference code(python HDNet_Inference.py),I get some errors.

Original stack trace for 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal':
File "HDNet_Inference.py", line 62, in
out2_1 = hourglass_refinement(x1,True)
File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 122, in hourglass_refinement
out0_d = hourglass_stack_no_incep(netIN)
File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 77, in hourglass_stack_no_incep
c0 = conv_layer(stack_in,layer_name('conv'),KER_SZ,NUM_CH[0],bn=bn,training=training)
File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 37, in conv_layer
kernel = variable('weights', [kernel_size, kernel_size, input_channels, output_channels], initializer, regularizer=tf.contrib.layers.l2_regularizer(0.0005))
File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 31, in variable
return tf.get_variable(name, shape, initializer=initializer, regularizer=regularizer, dtype=tf.float32, trainable=True)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 1500, in get_variable
aggregation=aggregation)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 1243, in get_variable
aggregation=aggregation)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 567, in get_variable
aggregation=aggregation)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 519, in _true_getter
aggregation=aggregation)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 933, in _get_single_variable
aggregation=aggregation)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 2519, in default_variable_creator
shape=shape)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1688, in init
shape=shape)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1818, in _init_from_args
initial_value(), name="initial_value", dtype=dtype)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 905, in
partition_info=partition_info)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/initializers.py", line 150, in _initializer
seed=seed)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/random_ops.py", line 175, in truncated_normal
shape_tensor, dtype, seed=seed1, seed2=seed2)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_random_ops.py", line 1016, in truncated_normal
name=name)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

I also try adding these lines of code immediately after importing Tensorflow. But there's no effect.

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)

How can I solve this problem?
Thanks a lot!

in theory, based on this, is it possible to create an account registrar with the ability to interact with live users?

Hi Yasamin,

Thanks for sharing your great work. May you please point me out to the line by which you've implements Eq. (4).

Thanks,

Hello, first of all congratulations on this great work.
I would like to know how you generated to 3d mesh from tang. et. al. groundtruth depth file as in the following figure

:

Hello, thank you for sharing this amazing work.

HDNet (trained by 34,000 RenderedPeople images and 100,000 Tiktok images) performance is very impressive compared to previous works :)

After reading the paper, I became curious about the performance of HDNet trained without RenderedPeople and with fewer RenderedPeople

Do you have pre-trained HDNet of those cases? or the results?

Hello author!
I have a complex problem when running the code. I don't know if it is caused by insufficient video memory. Have you ever had a similar problem? The error that comes up is as follows:

Traceback (most recent call last):
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
return fn(*args)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: tensorflow/core/kernels/cuda_solvers.cc:628: cuSolverDN call failed with status =6
[[{{node Svd_20}}]]
[[truediv_11/_1169]]
(1) Internal: tensorflow/core/kernels/cuda_solvers.cc:628: cuSolverDN call failed with status =6
[[{{node Svd_20}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "training_HDNet.py", line 195, in
i_limit:i_limit_tk})
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
run_metadata_ptr)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
feed_dict_tensor, options, run_metadata)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
run_metadata)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/client
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: tensorflow/core/kernels/cuda_solvers.cc:628: cuSolverDN call failed with status =6
[[node Svd_20 (defined at /home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/
[[truediv_11/_1169]]
(1) Internal: tensorflow/core/kernels/cuda_solvers.cc:628: cuSolverDN call failed with status =6
[[node Svd_20 (defined at /home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/
0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node Svd_20:
MatMul_93 (defined at /home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspo

Input Source operations connected to node Svd_20:
MatMul_93 (defined at /home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspo

Original stack trace for 'Svd_20':
File "training_HDNet.py", line 120, in
C,R,Rt,cen,K,Ki,origin,scaling)
File "/home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspose_transform_fu
PC2p, PC1_2 = transform_depth_PCs_dp_based2(C,R,Rt,cen,K,Ki,origin,scaling,d_i,d_j,i_r1_c1_r2_c
File "/home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspose_transform_fu
p2p, p1_2 = part_transformation2(i_limit,PC1,PC2,20); PC2p = tf.concat([PC2p,p2p],0); PC1_2 = t
File "/home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspose_transform_fu
,,p1_2 = get_pc_transformation2(p1,p2)
File "/home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspose_transform_fu
R,t = rigid_transform_3D(p1, p2)
File "/home/laihuaijing/liqianlin/HDNet_TikTok/training/training_code/utils/denspose_transform_fu
S, U, V = tf.linalg.svd(H)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/ops/li
tensor, compute_uv=compute_uv, full_matrices=full_matrices, name=name)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/ops/ge
full_matrices=full_matrices, name=name)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/framew
op_def=op_def)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/util/d
return func(*args, **kwargs)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/framew
op_def=op_def)
File "/home/laihuaijing/anaconda3/envs/HDNET/lib/python3.7/site-packages/tensorflow/python/framew
self._traceback = tf_stack.extract_stack()

It keeps saying "Sorry, you can't view or download this file at this time. Too many users have viewed or downloaded this file recently... etc.", can we have more alternative links/mirrors or a torrent please? Thank you very much for sharing this.

Hi yasamin, thanks to your great work, and I've really enjoyed it!

Here is my question.
I want to run your code on my images taken with my phone, but there is no document or requirement to make UV map.
Do I just use the output uv map of DensePose for the input of test code, or is there something other process I need to do?

Thank you for your answer in advance :)

Hi Yasamin, very awesome work!

I'm trying to train this work on my custom dataset. I found that the correspondence *.txts given in the training example are not simply defined by the UV ordinates obtained from the Densepose. If I directly construct correspondence between pixels with the same UV ordinates, it will lead to a very sparse mapping which is not as good as your pre-processed data. So I'd like to ask how did you construct the correspondence mapping between two densepose results?

Many Thanks!

Hi, I am also getting this result when running in the docker environment. @gexahedron Have you solved the problem?
This is the folder structure. Thanks.

...
├── HDNet_Inference.py
├── hourglass_net_depth.py
├── hourglass_net_normal.py
├── model
│   ├── depth_prediction
│   │   └── model_1920000
│   │       ├── checkpoint
│   │       ├── model_1920000.ckpt.data-00000-of-00001
│   │       ├── model_1920000.ckpt.index
│   │       └── model_1920000.ckpt.meta
│   └── normal_prediction
│       └── model_1710000
│           ├── checkpoint
│           ├── model_1710000.ckpt.data-00000-of-00001
│           ├── model_1710000.ckpt.index
│           └── model_1710000.ckpt.meta
...

Originally posted by @XiminLin in #9 (comment)

Hi!

I've managed to run your code in the docker image that you provided. However, I get a very weird output on your default image:

Do you know, what could be the problem?

Also, the running time was 22 minutes for a single 256x256 image, is it expected to run so slow?

Thank you for sharing the project---very impressive work.

I have two quick questions. First, in the paper it says that the network takes two inputs; the original image and the human figure mask. But on Github, a third input of the denspose estimate is suggested. Can you please clarify? Second, what is the runtime like? how long does it take for the network to process one image?

Thank you

After taking out the normal image and playing it continuously, there will be jitter. Is there any way to eliminate or reduce this?

The training data might could have different Ks, which is dependent on capture devices, cameras on cell phones. It seems to me that mean multiple K matrices( maybe even distortion coeffs) would be hard-coded to the network in the process. How does it affect the result? What are your thoughts about decoupling the intrinsics from this en/decoding network? Can intrinsics be provided when inference?

Tremendous work! Best paper in 2021. The output from Densepose is different from the one in the example. How is the uv 0-1 mapped to RGB? What configuration do you use when using Densepose? Also, is it really uv map or vertex map on the SMPL model I should be using? The uv map from this repo and the one genereated from Densepose by myself look different.

original

Yours

Mine

Hello, thanks for sharing this code.

I tried to run the demo but I failed building the environment. I use the docker as tensorflow doesn't match but I still have some issues. I used python 3.5 (because of the docker), Cuda 11.0 and Cudnn 10.2.

It is the first time I use docker and Cudnn and I'm not sure how to proceed to install the versions of CUDA and Cudnn you used to run the project.

The error message is the following one.

W0816 09:34:40.781714 140426061592320 deprecation_wrapper.py:119] From HDNet_Inference.py:15: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0816 09:34:40.781880 140426061592320 deprecation_wrapper.py:119] From HDNet_Inference.py:15: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

------------------------------
Hourglass Architecture
------------------------------
------------------------------
Hourglass Architecture
------------------------------
Model DR restored.
Model NP restored.
Processing file:  0043
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node hourglass_normal_prediction/conv1/Conv2D}}]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node hourglass_normal_prediction/conv1/Conv2D}}]]
	 [[hourglass_normal_prediction/conv18/BiasAdd/_177]]
0 successful operations.
0 derived errors ignored.

Thanks in advance.