Expressive Body Capture: 3D Hands, Face, and Body from a Single Image

Hello,

Very impressive works! I am wondering how can I extract the 3D joint coordinates from the code? (and also the name of the corresponding joint)

I am trying this model_output = body_model(return_verts=True, body_pose=body_pose) joints = model_output.joints.detach().cpu().numpy().squeeze() and it returns a (118,3) arrays. How can I locate 3D joints coordinates from this (e.g., left-elbow, left-wrist....)

Thanks!!!

Hi, I have this problem that I can not solve the file keypoints was created with OpenPose and is present in the directory, I do not understand the reason for this error?

SMPLIFY-X) bash-3.2$ python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder DATA_FOLDER --output_folder OUTPUT_FOLDER --visualize="True" --model_folder models --vposer_ckpt models/vposer_v1_0
Traceback (most recent call last):
  File "smplifyx/main.py", line 272, in <module>
    main(**args)
  File "smplifyx/main.py", line 201, in main
    for idx, data in enumerate(dataset_obj):
  File "/Users/piero/Desktop/SMPLIFY-X/smplify-x/smplifyx/data_parser.py", line 206, in __next__
    return self.next()
  File "/Users/piero/Desktop/SMPLIFY-X/smplify-x/smplifyx/data_parser.py", line 215, in next
    return self.read_item(img_path)
  File "/Users/piero/Desktop/SMPLIFY-X/smplify-x/smplifyx/data_parser.py", line 185, in read_item
    use_face_contour=self.use_face_contour)
  File "/Users/piero/Desktop/SMPLIFY-X/smplify-x/smplifyx/data_parser.py", line 54, in read_keypoints
    with open(keypoint_fn) as keypoint_file:
FileNotFoundError: [Errno 2] No such file or directory: 'DATA_FOLDER/keypoints/COCO_val2014_000000000459_rendered_keypoints.json'

I am using python3.6 , torch 1.1.0 and when i am running the smplifyx/main.py , as python3 smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder input/ --output_folder output/ --visualize="True" --model_folder ../models/ --vposer_ckpt ../vposer/ --part_segm_fn ../smplx_parts_segm.pkl

i am getting error, pyglet.gl.ContextException: Could not create GL context

and the code is getting stuck like this.

Any help?

Hello! Thanks for the great work! I want to know why number of parameters of "body_pose" is 32 ? and how to change "32" to "63" ? Can I get 63 parameters of "body_pose" directly from the smplifyx program？ And in the following code left_hand_pose = torch.einsum( 'bi,ij->bj', [left_hand_pose, self.left_hand_components]) left_hand_pose.shape : (1, 12) self.left_hand_components : (45, 45) Does left_hand_pose need to go through some processing？ Thank you!

Hi,

In the file fit_single_frame.py the code wants to import load_vposer from human_body_prior.tools.model_loader

However, there does not seem to be a load_vposer() function in human_body_prior/tools/model_loader.py. What am I missing here?

Good morning, this model seems to me extremely exciting!

I have configuration problems, in fact I have this error and I do not understand, keypoints should be generated by myself or should I do something?

This is the error:

python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder DATA_FOLDER --output_folder OUTPUT_FOLDER --visualize="True" --model_folder MODEL_FOLDER --vposer_ckpt VPOSER_FOLDER --part_segm_fn smplx_parts_segm.pkl Traceback (most recent call last): File "smplifyx/main.py", line 272, in main(args) File "smplifyx/main.py", line 201, in main for idx, data in enumerate(dataset_obj): File "/Users/piero******/anaconda3/envs/AVATAR/smplify-x/smplifyx/data_parser.py", line 206, in next return self.next() File "/Users/piero********/anaconda3/envs/AVATAR/smplify-x/smplifyx/data_parser.py", line 215, in next return self.read_item(img_path) File "/Users/piero********/anaconda3/envs/AVATAR/smplify-x/smplifyx/data_parser.py", line 185, in read_item use_face_contour=self.use_face_contour) File "/Users/piero********/anaconda3/envs/AVATAR/smplify-x/smplifyx/data_parser.py", line 54, in read_keypoints with open(keypoint_fn) as keypoint_file: FileNotFoundError: [Errno 2] No such file or directory: 'DATA_FOLDER/keypoints/76_img_keypoints.json'

Hi, @vchoutas! First of all, thank you for such nice work and clear code!

I want to fit SMPL-X model to images, where some joints are not visible, e.g. upper-body photos or even face photos. I tried several images and found out that the method works much worse when some joints are out of the frame (you can see examples below). The resulting fits don't much the input at all.

I believe this issue can be somehow fixed by nullifying some joints' weights during optimization or by some other optimization tweak. What do you think is the best way to do it?

Input | SMPLify-X :-------------------------:|:-------------------------: | | |

Hi, I just wanted to note this is amazing work! I've been trying to test Smplify-X on a custom image of a person standing up.

I've been getting this error: Traceback (most recent call last): File "smplifyx/render_results.py", line 99, in <module> mesh_fn = key_handler.get_mesh_fn() File "smplifyx/render_results.py", line 54, in get_mesh_fn return self.mesh_fns[self.idx] IndexError: list index out of range

Any idea how to fix this? Thanks!

Testing sample images found in smplify-x/src/homogenous-master/samples. Every time there is a segmentation fault at beginning of Stage 3:

~/smplify-x$ python3 smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder data --output_folder output --visualize="True" --model_folder ../smpl-x/models --vposer_ckpt ../smpl-x/vposer_v1_0

Processing: data/images/im1097.jpg Found Trained Model: ../smpl-x/vposer_v1_0/snapshots/TR00_E096.pt Camera initialization done after 7.6183 Camera initialization final loss 1101.1987 Stage 000 done after 28.3231 seconds
Stage 001 done after 5.3000 seconds
Stage 002 done after 13.7803 seconds
Orientation: 0%| | 0/1 [00:47<?, ?it/s Segmentation fault (core dumped)███▊ | 3/5 [00:47<00:38, 19.13s/it]

Hi I got an error at stage002 as following:

I debugged and found the error is at

 final_loss_val = monitor.run_fitting(
                    body_optimizer,
                    closure, final_params,
                    body_model,
                    pose_embedding=pose_embedding, vposer=vposer,
                    use_vposer=use_vposer)

Does anyone have any ideas what is wrong? Thx.

Hi,

I was able to run the smplify-x.

step 01 below is my image.

step 02 I generate the keypoints from openpose and generate the json file as follows.(plz remove the .pdf and add .json ) tummy_man_keypoints.pdf

step 03 Then i provide the step 01 image and the step 02 tummy_man_keypoints.json to the smplify-x.

i run the smplifyx,

python3 smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder DATA_FOLDER --output_folder OUTPUT_FOLDER --visualize="False" --model_folder MODEL_FOLDER --vposer_ckpt VPOSER_FOLDER

inside DATA_FOLDER contain two subfolders, images, where the images are located, and keypoints, where the OpenPose output should be stored.

inside MODEL_FOLDER put the SMPL-X Model

inside VPOSER_FOLDER put the VPoser: Variational Human Pose Prior

OUTPUT OF THE CONSOLE

Processing: /home/xxxx/simplifyx/smplify-x/DATA_FOLDER/images/tummy_man.png /home/xxxx/simplifyx/smplify-x/OUTPUT_FOLDER/meshes/tummy_man /home/xxxx/simplifyx/smplify-x/OUTPUT_FOLDER/results/tummy_man Found Trained Model: /home/xxxx/simplifyx/smplify-x/VPOSER_FOLDER/snapshots/TR00_E096.pt Camera initialization done after 2.4678 Camera initialization final loss 4296.7549 Stage 000 done after 5.3883 seconds
Stage 001 done after 3.4187 seconds
Stage 002 done after 11.2370 seconds
Stage 003 done after 9.5738 seconds
Stage 004 done after 8.8275 seconds
Body fitting Orientation 0 done after 38.4547 seconds
Body final loss val = 9400.93066
Orientation: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:38<00:00, 38.46s/it] Processing the data took: 00 hours, 00 minutes, 46 seconds

Then obj file was generated.The screenshot of the view is as follows.

Seems to be the output is wrong. Can any one help me to get the correct output?

Hi there, I've been having some issues with getting smplify-x to run and would appreciate any advice on the matter. The only other thread I've come across suggested installing and unpacking smplx in the right folders for neutral, male and female models, which I've done under smplify-x/data/models.

When I attempt to run: (tf) elizabeth@mili:~/smplify-x$ python smplifyx/main.py --config cfg_files/fit_smpl.yaml

Traceback (most recent call last): File "/home/elizabeth/smplify-x/smplifyx/main.py", line 272, in main(**args) File "/home/elizabeth/smplify-x/smplifyx/main.py", line 117, in main male_model = smplx.create(gender='male', **model_params) File "/home/elizabeth/miniconda3/envs/tf/lib/python3.9/site-packages/smplx/body_models.py", line 2410, in create raise ValueError(f'Unknown model type {model_type}, exiting!') ValueError: Unknown model type models, exiting!

Another error I've gotten is when I try the command: (tf) elizabeth@mili:~/smplify-x$ python smplifyx/main.py --config cfg_files/fit_smplx.yaml --data_folder DATA_FOLDER --output_folder OUTPUT_FOLDER --model_folder MODEL_FOLDER --vposer_ckpt VPOSER_FOLDER --part_segm_fn smplx_parts_segm.pkl

Traceback (most recent call last): File "/home/elizabeth/smplify-x/smplifyx/main.py", line 272, in main(**args) File "/home/elizabeth/smplify-x/smplifyx/main.py", line 84, in main dataset_obj = create_dataset(img_folder=img_folder, **args) File "/home/elizabeth/smplify-x/smplifyx/data_parser.py", line 47, in create_dataset return OpenPose(data_folder, **kwargs) File "/home/elizabeth/smplify-x/smplifyx/data_parser.py", line 136, in init for img_fn in os.listdir(self.img_folder) FileNotFoundError: [Errno 2] No such file or directory: 'DATA_FOLDER/images'

Thank you for your time & I'd greatly appreciate any help I could receive on this matter!

In the way we are running SMPLify, it reestimates shape and camera position with every frame. In our datasets, we know that it was the same person (same shape) and that the camera position was fixed. Could we estimate the shape and the camera from the whole session (e.g., 5-min. video), then fix it and run through the video again and estimate only the poses?

Hi, thank you for the amazing work. How can we fit the smplify-x on 3d scan data? There is a script for fitting FLAME to scan. Can we use this to fit smplify-x on 3d scan? FLAME fit to scan

When I run the result generated by the EHF dataset, except the first generated mesh is a normal human body shape, the other generated meshes are all alien shapes, why? Has anyone encountered this problem?

I took the smplx_texture_m_alb.png texture image file (see image below) from the SMPLX Blender Add-On. I wanted to add texture to the SMPLX mesh using this image outside of Blender.

Following the documentation of MPI Mesh, I used the following command: Mesh.set_texture_image('smplx_texture_m_alb.png'). However, I am facing an issue with the eye positioning in the output (see figure below):

What is causing this issue? Note that the texture is getting loaded correctly when imported in Blender add-on. (see figure below)