Hi Yasamin, awesome work! I'm trying to train the model on my own dataset, and I wonder if you could clarify the formats of the labeled data. Specifically, for normal, what should the channel 1,2,3 of the txt files each store? And how did you convert them into visualization?

Hello, first of all congratulations on this great work. I would like to know how you generated to 3d mesh from tang. et. al. groundtruth depth file as in the following figure :

Hello, thanks for sharing this code.

I tried to run the demo but I failed building the environment. I use the docker as tensorflow doesn't match but I still have some issues. I used python 3.5 (because of the docker), Cuda 11.0 and Cudnn 10.2.

It is the first time I use docker and Cudnn and I'm not sure how to proceed to install the versions of CUDA and Cudnn you used to run the project.

The error message is the following one.

W0816 09:34:40.781714 140426061592320 deprecation_wrapper.py:119] From HDNet_Inference.py:15: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.

W0816 09:34:40.781880 140426061592320 deprecation_wrapper.py:119] From HDNet_Inference.py:15: The name tf.logging.ERROR is deprecated. Please use tf.compat.v1.logging.ERROR instead.

------------------------------
Hourglass Architecture
------------------------------
------------------------------
Hourglass Architecture
------------------------------
Model DR restored.
Model NP restored.
Processing file:  0043
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.5/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node hourglass_normal_prediction/conv1/Conv2D}}]]
  (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
	 [[{{node hourglass_normal_prediction/conv1/Conv2D}}]]
	 [[hourglass_normal_prediction/conv18/BiasAdd/_177]]
0 successful operations.
0 derived errors ignored.

Thanks in advance.

Hi yasamin, thanks to your great work, and I've really enjoyed it!

Here is my question. I want to run your code on my images taken with my phone, but there is no document or requirement to make UV map. Do I just use the output uv map of DensePose for the input of test code, or is there something other process I need to do?

Thank you for your answer in advance :)

It keeps saying "Sorry, you can't view or download this file at this time. Too many users have viewed or downloaded this file recently... etc.", can we have more alternative links/mirrors or a torrent please? Thank you very much for sharing this.

Not sure what is going on here. I am running the new NVIDIA 3080 TI. It says my GPU is not supported which I have never had happen to me before even on older codebases. :/ ` Traceback (most recent call last): File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in do_call return fn(*args) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1339, in run_fn self.extend_graph() File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1374, in extend_graph tf_session.ExtendSession(self.session) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal: Could not satisfy explicit device specification '' because the node {{colocation_node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal}} was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[] L2Loss: CPU XLA_CPU Assign: CPU VariableV2: CPU Identity: CPU XLA_CPU Add: CPU XLA_CPU Mul: CPU XLA_CPU TruncatedNormal: CPU XLA_CPU Const: CPU XLA_CPU

Colocation members, user-requested devices, and framework assigned devices, if any: hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/shape (Const) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mean (Const) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/stddev (Const) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (TruncatedNormal) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mul (Mul) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal (Add) hourglass_stack_fused_depth_prediction/conv1/weights (VariableV2) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Assign (Assign) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/read (Identity) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/scale (Const) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/L2Loss (L2Loss) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer (Mul) /device:GPU:0

 [[{{node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "HDNet_Inference.py", line 64, in tf.global_variables_initializer().run() File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2679, in run run_using_default_session(self, feed_dict, self.graph, session) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 5614, in run_using_default_session session.run(operation, feed_dict) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in run feed_dict_tensor, options, run_metadata) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in do_run run_metadata) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal: Could not satisfy explicit device specification '' because the node node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (defined at /home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:28) placed on device Device assignments active during op 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal' creation: with tf.device(None): </home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py:1696> with tf.device(/gpu:0): </home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:26> was colocated with a group of nodes that required incompatible device '/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0]. Colocation Debug Info: Colocation group had the following types and supported devices: Root Member(assigned_device_name_index=-1 requested_device_name='/device:GPU:0' assigned_device_name='' resource_device_name='/device:GPU:0' supported_device_types=[CPU] possible_devices_=[] L2Loss: CPU XLA_CPU Assign: CPU VariableV2: CPU Identity: CPU XLA_CPU Add: CPU XLA_CPU Mul: CPU XLA_CPU TruncatedNormal: CPU XLA_CPU Const: CPU XLA_CPU

Colocation members, user-requested devices, and framework assigned devices, if any: hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/shape (Const) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mean (Const) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/stddev (Const) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (TruncatedNormal) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/mul (Mul) hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal (Add) hourglass_stack_fused_depth_prediction/conv1/weights (VariableV2) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Assign (Assign) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/read (Identity) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/scale (Const) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer/L2Loss (L2Loss) /device:GPU:0 hourglass_stack_fused_depth_prediction/conv1/weights/Regularizer/l2_regularizer (Mul) /device:GPU:0

 [[node hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal (defined at /home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:28) ]]Additional information about colocations:No node-device colocations were active during op 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal' creation.

Device assignments active during op 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal' creation: with tf.device(None): </home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py:1696> with tf.device(/gpu:0): </home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py:26>

Original stack trace for 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal': File "HDNet_Inference.py", line 57, in out2_1 = hourglass_refinement(x1,True) File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 119, in hourglass_refinement out0_d = hourglass_stack_no_incep(netIN) File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 74, in hourglass_stack_no_incep c0 = conv_layer(stack_in,layer_name('conv'),KER_SZ,NUM_CH[0],bn=bn,training=training) File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 34, in conv_layer kernel = variable('weights', [kernel_size, kernel_size, input_channels, output_channels], initializer, regularizer=tf.contrib.layers.l2_regularizer(0.0005)) File "/home/killab/ml/models/ShapeGeneration/HDNet_TikTok/hourglass_net_depth.py", line 28, in variable return tf.get_variable(name, shape, initializer=initializer, regularizer=regularizer, dtype=tf.float32, trainable=True) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1496, in get_variable aggregation=aggregation) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 1239, in get_variable aggregation=aggregation) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 562, in get_variable aggregation=aggregation) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 514, in _true_getter aggregation=aggregation) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 929, in _get_single_variable aggregation=aggregation) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 259, in call return cls._variable_v1_call(*args, **kwargs) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 220, in _variable_v1_call shape=shape) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 198, in previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2511, in default_variable_creator shape=shape) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 263, in call return super(VariableMetaclass, cls).call(*args, **kwargs) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1568, in init shape=shape) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variables.py", line 1698, in _init_from_args initial_value(), name="initial_value", dtype=dtype) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/variable_scope.py", line 901, in partition_info=partition_info) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/contrib/layers/python/layers/initializers.py", line 150, in _initializer seed=seed) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 178, in truncated_normal shape_tensor, dtype, seed=seed1, seed2=seed2) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 1013, in truncated_normal name=name) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op op_def=op_def) File "/home/killab/anaconda3/envs/HDNet/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init self._traceback = tf_stack.extract_stack()

`

Thank you for sharing the project---very impressive work.

I have two quick questions. First, in the paper it says that the network takes two inputs; the original image and the human figure mask. But on Github, a third input of the denspose estimate is suggested. Can you please clarify? Second, what is the runtime like? how long does it take for the network to process one image?

Thank you

I'm interested in the new view video generated with the estimated depth, would you please share some methods to do this. Again, your work is really amazing!

Hi yasamin,thanks to your wonderful works.

When I run your Inference code(python HDNet_Inference.py),I get some errors.

Original stack trace for 'hourglass_stack_fused_depth_prediction/conv1/weights/Initializer/truncated_normal/TruncatedNormal': File "HDNet_Inference.py", line 62, in out2_1 = hourglass_refinement(x1,True) File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 122, in hourglass_refinement out0_d = hourglass_stack_no_incep(netIN) File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 77, in hourglass_stack_no_incep c0 = conv_layer(stack_in,layer_name('conv'),KER_SZ,NUM_CH[0],bn=bn,training=training) File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 37, in conv_layer kernel = variable('weights', [kernel_size, kernel_size, input_channels, output_channels], initializer, regularizer=tf.contrib.layers.l2_regularizer(0.0005)) File "/home/schuang/rti3_surreal/HDNet_TikTok/hourglass_net_depth.py", line 31, in variable return tf.get_variable(name, shape, initializer=initializer, regularizer=regularizer, dtype=tf.float32, trainable=True) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 1500, in get_variable aggregation=aggregation) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 1243, in get_variable aggregation=aggregation) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 567, in get_variable aggregation=aggregation) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 519, in _true_getter aggregation=aggregation) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 933, in _get_single_variable aggregation=aggregation) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 258, in call return cls._variable_v1_call(*args, **kwargs) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call shape=shape) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 197, in previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 2519, in default_variable_creator shape=shape) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 262, in call return super(VariableMetaclass, cls).call(*args, **kwargs) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1688, in init shape=shape) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variables.py", line 1818, in _init_from_args initial_value(), name="initial_value", dtype=dtype) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py", line 905, in partition_info=partition_info) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/initializers.py", line 150, in _initializer seed=seed) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/random_ops.py", line 175, in truncated_normal shape_tensor, dtype, seed=seed1, seed2=seed2) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_random_ops.py", line 1016, in truncated_normal name=name) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper op_def=op_def) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op attrs, op_def, compute_device) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal op_def=op_def) File "/home/schuang/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init self._traceback = tf_stack.extract_stack()

I also try adding these lines of code immediately after importing Tensorflow. But there's no effect.

from tensorflow.compat.v1 import ConfigProto from tensorflow.compat.v1 import InteractiveSession config = ConfigProto() config.gpu_options.allow_growth = True session = InteractiveSession(config=config)

How can I solve this problem? Thanks a lot!

Tremendous work! Best paper in 2021. The output from Densepose is different from the one in the example. How is the uv 0-1 mapped to RGB? What configuration do you use when using Densepose? Also, is it really uv map or vertex map on the SMPL model I should be using? The uv map from this repo and the one genereated from Densepose by myself look different.

original Yours Mine

When I change the variable BATCH_SIZE in training_HDNet.py, it raises an error in Depth2Points3D_transformed_vector. The issue is there is an another batch_size parameter in the line 51 of the same function. Even if I set it to be the same as BATCH_SIZE, the training wouldn't work as intended because, tiktok batches are discarded in transform_depth_PCs_dp_based2. Since each TikTok sample will have different number of correspondences, I guess the best solution is iterating over batches inside transform_depth_PCs_dp_based2.

Hi, Thank you for sharing this great work! Do you have any ideas for future work such as fusion multi-view of mesh results, and Bust-Waist-Hip measurements based on these achievements? I'm really interested in this future work. Please notify me if you publish a corresponding paper/repository in this field.

Hi, I am also getting this result when running in the docker environment. @gexahedron Have you solved the problem? This is the folder structure. Thanks.

...
├── HDNet_Inference.py
├── hourglass_net_depth.py
├── hourglass_net_normal.py
├── model
│   ├── depth_prediction
│   │   └── model_1920000
│   │       ├── checkpoint
│   │       ├── model_1920000.ckpt.data-00000-of-00001
│   │       ├── model_1920000.ckpt.index
│   │       └── model_1920000.ckpt.meta
│   └── normal_prediction
│       └── model_1710000
│           ├── checkpoint
│           ├── model_1710000.ckpt.data-00000-of-00001
│           ├── model_1710000.ckpt.index
│           └── model_1710000.ckpt.meta
...

Originally posted by @XiminLin in https://github.com/yasaminjafarian/HDNet_TikTok/issues/9#issuecomment-1243054092