Hi, Thanks for your great work and open source code! Sorry to disturb you. I have trouble with how to preprocess the 'Florence' dataset and calculate the point-to-plane RMSE. I have made some effort but still do not deal with this. I would be very grateful if you can share this part of code with me. Looking forward to hearing from you. Thank you!

Can you help me？ Thank you~ There is an error in step 4:

1. ./runtests.sh # -D_GLIBCXX_USE_CXX11_ABI=0 to -D_GLIBCXX_USE_CXX11_ABI=1
2. bazel build ...
3. copy MGCNet/tf_mesh_renderer/bazel-out/k8-fastbuild/bin/mesh_renderer/kernels/rasterize_triangles_kernel.so to MGCNet/thirdParty/tf_mesh_renderer/mesh_renderer/kernels
4. python test_images.py

NOTE: tensorflow-gpu1.13.1 ubuntu18.04

ERROR: Connected to pydev debugger (build 201.7223.92) /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /home/george/pycharm/face_3d/MGCNet Finish copy /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:54: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. pt_mean = self.hdf5io_pt_model.GetValue('mean').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:58: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. pt_pcaBasis = self.hdf5io_pt_model.GetValue('pcaBasis').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:61: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. pt_pcaVariance = self.hdf5io_pt_model.GetValue('pcaVariance').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:76: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. rgb_mean = self.hdf5io_rgb_model.GetValue('mean').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:80: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. rgb_pcaBasis = self.hdf5io_rgb_model.GetValue('pcaBasis').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:83: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. rgb_pcaVariance = self.hdf5io_rgb_model.GetValue('pcaVariance').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:89: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. uv = self.hdf5io_rgb_model.GetValue('uv').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:133: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. exp_pcaBasis = self.hdf5io_exp_model.GetValue('pcaBasis').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:136: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. exp_pcaVariance = self.hdf5io_exp_model.GetValue('pcaVariance').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:145: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. mesh_tri_reference = self.hdf5io_pt_representer.GetValue('tri').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:153: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. idx_subTopo = self.hdf5io_pt_representer.GetValue('idx_sub').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:158: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. self.nplist_v_ring_f_flat_np = self.hdf5io_pt_representer.GetValue('vertex_ring_face_flat').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:160: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. self.nplist_ver_ref_face_num = self.hdf5io_pt_representer.GetValue('vertex_ring_face_num').value /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:236: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray return np.array(list_v_ring_f), np.array(list_v_ring_f_index) /home/george/pycharm/face_3d/MGCNet/src_common/geometry/gpmm/bfm09_tf_uv.py:167: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead. idx_lm68_np = self.hdf5io_pt_representer.GetValue('idx_lm68').value WARNING:tensorflow:From /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version. Instructions for updating: Colocations handled automatically by placer. WARNING:tensorflow:From /home/george/pycharm/face_3d/MGCNet/src_common/geometry/render/lighting.py:151: calling norm (from tensorflow.python.ops.linalg_ops) with keep_dims is deprecated and will be removed in a future version. Instructions for updating: keep_dims is deprecated, use keepdims instead WARNING:tensorflow:From /home/george/pycharm/face_3d/MGCNet/src_common/geometry/render/api_tf_mesh_render.py:266: calling l2_normalize (from tensorflow.python.ops.nn_impl) with dim is deprecated and will be removed in a future version. Instructions for updating: dim is deprecated, use axis instead WARNING:tensorflow:From /home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. Traceback (most recent call last): File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InvalidArgumentError: RasterizeTriangles expects vertices to have shape (-1, 4). [[{{node RasterizeTriangles_12}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/george/.conda/envs/cuda10/lib/python3.7/contextlib.py", line 130, in exit self.gen.throw(type, value, traceback) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 5253, in get_controller yield g File "/home/george/pycharm/face_3d/MGCNet/mytest_image.py", line 131, in pred = system.inference(sess, image_rgb_b) File "/home/george/pycharm/face_3d/MGCNet/src_tfGraph/build_graph.py", line 1103, in inference results = sess.run(fetches, feed_dict={'pl_input:0':inputs}) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 929, in run run_metadata_ptr) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run feed_dict_tensor, options, run_metadata) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run run_metadata) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: RasterizeTriangles expects vertices to have shape (-1, 4). [[node RasterizeTriangles_12 (defined at :223) ]]

Caused by op 'RasterizeTriangles_12', defined at: File "/opt/pycharm-2020.1.1/plugins/python/helpers/pydev/pydevd.py", line 2131, in main() File "/opt/pycharm-2020.1.1/plugins/python/helpers/pydev/pydevd.py", line 2122, in main globals = debugger.run(setup['file'], None, None, is_module) File "/opt/pycharm-2020.1.1/plugins/python/helpers/pydev/pydevd.py", line 1431, in run return self._exec(is_module, entry_point_fn, module_name, file, globals, locals) File "/opt/pycharm-2020.1.1/plugins/python/helpers/pydev/pydevd.py", line 1438, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "/opt/pycharm-2020.1.1/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/home/george/pycharm/face_3d/MGCNet/mytest_image.py", line 88, in FLAGS, img_height=FLAGS.img_height, img_width=FLAGS.img_width, batch_size=FLAGS.batch_size File "/home/george/pycharm/face_3d/MGCNet/src_tfGraph/build_graph.py", line 949, in build_test_graph self.build_testVisual_graph() File "/home/george/pycharm/face_3d/MGCNet/src_tfGraph/build_graph.py", line 1059, in build_testVisual_graph self.gpmm_frustrum, gpmm_tar_mv, gpmm_tar_eye, fore=opt.flag_fore, tone=False File "/home/george/pycharm/face_3d/MGCNet/src_tfGraph/deep_3dmm_decoder.py", line 267, in decoder_renderColorMesh mtx_perspect_frustrum, list_mtx_model_view[i], list_cam_position[i], tone File "/home/george/pycharm/face_3d/MGCNet/src_tfGraph/deep_3dmm_decoder.py", line 459, in gpmm_render_image mtx_perspect_frustrum, cam_position, opt.img_width, opt.img_height) File "/home/george/pycharm/face_3d/MGCNet/src_common/geometry/render/api_tf_mesh_render.py", line 99, in mesh_renderer_camera_light image_width, image_height, [-1] * vertex_attributes.shape[2].value) File "/home/george/pycharm/face_3d/MGCNet/thirdParty/tf_mesh_renderer/mesh_renderer/rasterize_triangles.py", line 125, in rasterize_triangles image_height)) File "", line 223, in rasterize_triangles File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/george/.conda/envs/cuda10/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in init self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): RasterizeTriangles expects vertices to have shape (-1, 4). [[node RasterizeTriangles_12 (defined at :223) ]]

Process finished with exit code 1

I followed the instructrions to build tf_mesh_renderer. I used bazel==4.1.0 downloaded from official github I used gcc==5.4.0 I used git submodule to fetch the tf_mesh_renderer code and run bazel build ... got the result as followed: Starting local Bazel server and connecting to it... INFO: Analyzed 10 targets (28 packages loaded, 336 targets configured). INFO: Found 10 targets... INFO: Elapsed time: 9.038s, Critical Path: 4.72s INFO: 19 processes: 12 internal, 7 linux-sandbox. INFO: Build completed successfully, 19 total actions

However, I build rasterize_triangles_kernel.so under path of tf_mesh_renderer/bazel-bin/mesh_renderer/kernels/rasterize_triangles_kernel.so instead of tf_mesh_renderer/mesh_renderer/kernels/rasterize_triangles_kernel.so as shown in the code.

The error is like: tensorflow.python.framework.errors_impl.NotFoundError: tf_mesh_renderer/bazel-bin/mesh_renderer/kernels/rasterize_triangles_kernel.so: undefined symbol: _ZN10tensorflow12OpDefBuilder4AttrESs

Could you please help me out here?

Hi!

I'm doing a personal project on top of your work with the provided pretrained weights and I really thank you for sharing your work.

I have tested test_image.py on an image from MultiPIE dataset, and the results are sometimes good, but sometimes bad. Here's a bad example.

Results of left and right images are generally good, but this problem is often observed for frontal images. I was wondering if the pretrained weights are not from a model that was trained with MultiPIE dataset.

I am planning to fine-tune the weights with MultiPIE dataset, but just needed to check if this would help.

Thank you.

when I use facenet as perception loss , this error happen: from align.py :line 110, in TransformFromPointsTF : tf.self_adjoint_eig(N) tensorflow.python.framework.errors_impl.InvalidArgumentError: Self-adjoint eighen decomposition was not successful , the input might not be valid. [[{node SelfAdjointEigV2}]] [[{{node gradients/Reshape_37_grad/Shape}}]]. my tf.version is 1.13.0

when I swith tf 1.13.1. there are another problem: InvalidArgumentError :Got info = 3 for batch index 0 , expected info=0, debug_info = heevd

I can't found any solution for this problem .can you help me ? Thank you !

I'm trying to use a custom dataset to train the model using train_unsupervised.py.

However, it seems that it needs a skin image (named as ..._skin.jpg).

How can I get those skin images from plain images?

Hi. Thanks for great work!

I have few questsions of dataset.

1. Is there any other dataset for training rather than 'Multi-Pie' and '300W-LP'?
2. How get ground-truth pose data (camera extrinsic, rotation/translation) for '300W-LP' datasets? Isn't it wild images ?

• I have not downloaded 'Multi-Pie' dataset, but I think this dataset might have gt pose data

Thank you.

Best regards, YJHong.

Hi! Good morning! I try to use three datasets(CelebA、300W-LP and LS3D-W balanced subset) to train the MGCNet with your code. But I got a much worse result compared with the pre-train model. As shown below: The results produced by the pre-train model is wonderful.

But the results from my reproduce model is ugly.

I followed the training command in "train_unsupervise.py" without changing any parameter except the batchsize=1. The command is as follow:

I think there are some problems during my reproduce: (1) The step is 20K, which is much smaller than 400K in the paper. (2) I do not get the Multi-PLE dataset, so the result should be worse than pre-train. (3) The weight of epipolar_weight was set to 0.00 in "train_unsupervise.py", I think it is a mistake. In the paper, this weight was set to 0.001. (4) I set the batch size to 1 for the limited GPU source. (5) Can the code support training with multiple GPUs?

Now I am trying to train again with the same hyper-parameters in the paper. Thanks for your answers!

First, I use 'conda install bazel==0.11.1' to install bazel 0.11.1. I use the default gcc of ubuntu 18.04.2 is 7.5.0. However, when I run 'Bazel build ...', I got these error: It seems that the bazel can not get the gcc file?

Then, I try to use 'conda install gcc==4.8.5' to install gcc 4.8.5. But after I install gcc 4.8.5, I can not import tensorflow. Here are the errors:

I really want to know how to install the bazel and gcc? Maybe I should not use conda to install them? Thanks for you help!

Hi. The "test_image.py" is used to inference a single unprocessed image. And I have a question: How to inference on a video directly? Since I want to get more frame consistency. If I inference these frames individually, there may have some jitter, and I want to know how to address this problem?

Hi, sorry to disturb you. I have several questions about the pose and projection matrix code.

1. why we need to add defined_pose_main to pred_pose_render?

2. what do the variables "list_gpmm_mv" and "list_gpmm_eye" do ? I found they are calculated from rotation matrix and translation vector, and used in mesh_renderer_camera_light(), but I don't know the meaning of them.

3. in project3d_batch(), why the projected landmarks need to be divided by 'pt_batch_homo_2d_w'

Bumps tensorflow-gpu from 1.13.1 to 2.9.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps numpy from 1.16.6 to 1.22.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps opencv-python from 3.4.2.17 to 4.2.0.32.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi, thank you for your work.

I already read your README, and you said: "I find many issue happens here, so let's make this more clear. The tf_mesh_render does not return triangle id for each pixel after rasterise, we do this by our self and add these changes as submodule to mgcnet.".

I have one question for that. Did you use another code instead of tf_mesh_render?

If yes, can you explain where's that code for the change ?

Thanks.

Traceback (most recent call last): File "/root/MGCNet/thirdParty/tf_mesh_renderer/mesh_renderer/rasterize_triangles.py", line 26, in os.path.join(os.environ['TEST_SRCDIR'], File "/opt/conda/lib/python3.6/os.py", line 669, in getitem raise KeyError(key) from None KeyError: 'TEST_SRCDIR' Does the value 'TEST_SRCDIR' need to be assigned to myself?How to do it？