First, thanks for your great work.

Just wondering if exporting animated format(such as bvh or fbx) is supported ? Looks like demo.py only exports T-pose skeleton . I've tried to combine all outputs to fbx format but had trouble with animating character with a given pose. (In blender) I used pose (FrameN, Joint_Num,3) data as an input of euler value but it doesn't work.

Any Suggestions??

Great work! When I try to visualize the animation via blender following this, an error occured.

Saved: '../demo/images/0000.png'
 Time: 00:00.28 (Saving: 00:00.25)

Traceback (most recent call last):
  File "/home/yusun/Documents/GitHub/neural-blend-shapes/blender_scripts/render_mesh.py", line 96, in <module>
    batch_render(args.obj_path)
  File "/home/yusun/Documents/GitHub/neural-blend-shapes/blender_scripts/render_mesh.py", line 56, in batch_render
    render_obj(os.path.join(dir, file))
  File "/home/yusun/Documents/GitHub/neural-blend-shapes/blender_scripts/render_mesh.py", line 42, in render_obj
    obj.select_set(True)
AttributeError: 'Object' object has no attribute 'select_set'

I am using Blender v2.91a. Any suggestion?

hello,I am a student of USTB,Thanks for excite project,I want to apply your project to maya, but I can't understand rig,I make a FBX file and run this code to get blendshape values,But with the wrong rig,the results is a bit bad, if you have made a fbx file?

miko_siggraph_neural_blend_shapes.zip

python demo.py --pose_file=./eval_constant/sequences/house-dance.npy --obj_path=./eval_constant/meshes/miko.obj --normalize=1 --obj_output=0 --animated_bvh=1

(neural-blend-shapes) D:\CG_Source\NeRFs\3D_Avatar_Pipeline\neural-blend-shapes>python demo.py --pose_file=.\eval_constant\sequences\greeting.npy --obj_path=.\eval_constant\meshes\test_remesh.obj --animated_bvh=1 --obj_output=0 --normalize=1
Traceback (most recent call last):
  File "demo.py", line 150, in <module>
    main()
  File "demo.py", line 132, in main
    env_model, res_model = load_model(device, model_args, topo_loader, args.model_path)
  File "demo.py", line 59, in load_model
    geo, att, gen = create_envelope_model(device, model_args, topo_loader, is_train=False, parents=parent_smpl)
  File "D:\CG_Source\NeRFs\3D_Avatar_Pipeline\neural-blend-shapes\architecture\__init__.py", line 34, in create_envelope_model
    save_freq=args.save_freq)
  File "D:\CG_Source\NeRFs\3D_Avatar_Pipeline\neural-blend-shapes\models\networks.py", line 54, in __init__
    topo_loader, requires_recorder, is_cont, save_freq)
  File "D:\CG_Source\NeRFs\3D_Avatar_Pipeline\neural-blend-shapes\models\meshcnn_base.py", line 56, in __init__
    val.extend([1 / len(neighbors)] * len(neighbors))
ZeroDivisionError: division by zero

I get this error, everytime I try your model with custom meshes. I checked to make sure they are triangulated and called with normalize=1 but this happens for all the meshes I have. Is there some MeshLab filter that I can apply/script which could parse my custom mesh before sending it to your model or a simple fix for this issue. Thank you for your amazing work!

Hi @PeizhuoLi, thanks for sharing this wonderful work ,and I have try some obj with different shapes, but encountered penetration problem. it seems this situation didn't discuss in the paper, can you give some advice? thanks~

Similar to the current cpu version. Provide an environment.yml for cuda based pytorch.

It is non trivial to switch between gpu and cpu. https://github.com/pytorch/pytorch/issues/30664.

Traceback (most recent call last): File "", line 1, in File "D:\360Downloads\neural_blend_shapes_main\blender_scripts\render_mesh.py", line 96, in batch_render(args.obj_path) File "D:\360Downloads\neural_blend_shapes_main\blender_scripts\render_mesh.py", line 56, in batch_render render_obj(os.path.join(dir, file)) File "D:\360Downloads\neural_blend_shapes_main\blender_scripts\render_mesh.py", line 33, in render_obj add_material_for_objs([obj], 'character') File "D:\360Downloads\neural_blend_shapes_main\blender_scripts\render_mesh.py", line 28, in add_material_for_objs raise Exception("This line shouldn't be reached") Exception: This line shouldn't be reached

I don't know why this happen. My command is ' blender -P render_mesh.py -b'

This script allows to export neural-blend-shapes file (.fbx, .glb)

To get neural-blendshapes model , just simply run blender -b -P nbs_fbx_output.py -- --input ../demo --output ../demo/output.fbx

To get envelop-only model , just simply run blender -b -P nbs_fbx_output.py -- --input ../demo --output ../demo/output.fbx --envelope_only 1

#Required Input direcotry is the directory where the ouputs are generated from 'demo.py'

'T-pose.obj', 'skeleton.bvh' , 'weight.npy' and 'basis.npy' , 'coff.npy' should exist in input directory (e.g. ../demo) If 'basis.npy' , 'coff.npy' don't exist , envelope-only model will be exported.

'basis.npy' and 'coff.npy' (that is required to get neural-blend-shape model) can be achieved from params called basis_full, coff respectively in ./architecture/blend_shapes.py

e.g. I simply add this code in the forward function in ./architecture/blend_shapes.py

import numpy as np np_basis = basis_full.detach().cpu().numpy() np_coff = coff.detach().cpu().numpy() np.save('./demo/basis.npy',np_basis) np.save('./demo/coff.npy',np_coff)

I met some similar issue as other mentioned before. I'm not sure if I missed some step to prepare the mesh. I tried to align it and fix the non manifold issue, but the mesh is still broken after I run the demo. I've attached my files, it contains the one I use as referral(maynard:Mesh) and the one I'm running the demo on(castle:Castleguard1), if you can take a look. Thanks!

Uploading confusion.fbx…

您好，我测试了一些自己的模型，增加了normalize=1，发现骨骼尺度上有差距，最终出来的蒙皮权重就很差，有几个问题想问下：

1. 三角化后的模型consistent upright and front facing orientation.是指模型导入引擎，朝向(比如fbx脚本里的axis_forward='-Z', axis_up='Y')一致，然后是T-pose就行？ 还是三角面片坐标上有什么含义？
2. 和smpl_std.obj对齐，直接使用您给定的归一化函数就可以？还是有什么更细致的要求？

Thank you for your great work. After reading the paper, I still have some questions. So I hope you or anyone else can answer me if possible.

In the paper, the Residual Deformation Branch learns to predict blendshapes for each individual character. I'm wondering how these blendshapes are defined.

I'm not familiar with body blendshapes but as far as I know, blendshapes for facial expressions like jawOpen, eyeBlinkLeft, smileRight, etc., are semantically defined. In the paper[1], personalized facial blendshapes are learned via blendshape gradient loss function, which forces each of the generated blendshapes to have specific semantic meaning.

Another way to use blendshapes for facial expression is like what was done in MetaHuman[2] (Unreal Engine), where expressions are produced by bones. Blendshapes (called morph targets in Unreal Engine) are used to refine the face, which add more details. I think this is more similar to your work.

So I would like to know some details on your blendshapes: how they are defined, how they are learned, etc.

I really appreciate it if you could answer my questions.

Ref: [1] Personalized Face Modeling for Improved Face Reconstruction and Motion Retargeting [2] MetaHuman

Hi! When I reproduced this great work and trained from scratch, the skeleton produced some weird result, as shown in the picture. The left is the result I reproduced, and the right is the result produced from the pre_trained model. I think it may because the end effector was set wrongly, but I didn't change any code about it. Could you please help me with this problem? Looking forward to your reply!

Hi. Could you tell how the sequence pose files(located in ./eval_constant/sequences/ *.npy) are generated ?? It seems like those files were converted from Mixamo files but couldn't find a clue to create those. If you don't mind , could you let us know this process ?

Is there a recommended way for converting an arbitrary posed mesh to a t-posed mesh?

Side Question - Also since you are able to calculate skinning weights of a mesh, why isn't it possible to do neural blend shapes on any arbitrary posed mesh?