Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set (CVPRW 2019). A PyTorch implementation.

Overview

Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set —— PyTorch implementation

This is an unofficial official pytorch implementation of the following paper:

Y. Deng, J. Yang, S. Xu, D. Chen, Y. Jia, and X. Tong, Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set, IEEE Computer Vision and Pattern Recognition Workshop (CVPRW) on Analysis and Modeling of Faces and Gestures (AMFG), 2019. (Best Paper Award!)

The method enforces a hybrid-level weakly-supervised training for CNN-based 3D face reconstruction. It is fast, accurate, and robust to pose and occlussions. It achieves state-of-the-art performance on multiple datasets such as FaceWarehouse, MICC Florence and NoW Challenge.

For the original tensorflow implementation, check this repo.

This implementation is written by S. Xu.

Performance

● Reconstruction accuracy

The pytorch implementation achieves lower shape reconstruction error (9% improvement) compare to the original tensorflow implementation. Quantitative evaluation (average shape errors in mm) on several benchmarks is as follows:

Method FaceWareHouse MICC Florence NoW Challenge
Deep3DFace Tensorflow 1.81±0.50 1.67±0.50 1.54±1.29
Deep3DFace PyTorch 1.64±0.50 1.53±0.45 1.41±1.21

The comparison result with state-of-the-art public 3D face reconstruction methods on the NoW face benchmark is as follows:

Rank Method Median(mm) Mean(mm) Std(mm)
1. DECA[Feng et al., SIGGRAPH 2021] 1.09 1.38 1.18
2. Deep3DFace PyTorch 1.11 1.41 1.21
3. RingNet [Sanyal et al., CVPR 2019] 1.21 1.53 1.31
4. Deep3DFace [Deng et al., CVPRW 2019] 1.23 1.54 1.29
5. 3DDFA-V2 [Guo et al., ECCV 2020] 1.23 1.57 1.39
6. MGCNet [Shang et al., ECCV 2020] 1.31 1.87 2.63
7. PRNet [Feng et al., ECCV 2018] 1.50 1.98 1.88
8. 3DMM-CNN [Tran et al., CVPR 2017] 1.84 2.33 2.05

For more details about the evaluation, check Now Challenge website.

● Visual quality

The pytorch implementation achieves better visual consistency with the input images compare to the original tensorflow version.

● Speed

The training speed is on par with the original tensorflow implementation. For more information, see here.

Major changes

● Differentiable renderer

We use Nvdiffrast which is a pytorch library that provides high-performance primitive operations for rasterization-based differentiable rendering. The original tensorflow implementation used tf_mesh_renderer instead.

● Face recognition model

We use Arcface, a state-of-the-art face recognition model, for perceptual loss computation. By contrast, the original tensorflow implementation used Facenet.

● Training configuration

Data augmentation is used in the training process which contains random image shifting, scaling, rotation, and flipping. We also enlarge the training batchsize from 5 to 32 to stablize the training process.

● Training data

We use an extra high quality face image dataset FFHQ to increase the diversity of training data.

Requirements

This implementation is only tested under Ubuntu environment with Nvidia GPUs and CUDA installed.

Installation

  1. Clone the repository and set up a conda environment with all dependencies as follows:
git clone https://github.com/sicxu/Deep3DFaceRecon_pytorch.git --recursive
cd Deep3DFaceRecon_pytorch
conda env create -f environment.yml
source activate deep3d_pytorch
  1. Install Nvdiffrast library:
cd nvdiffrast    # ./Deep3DFaceRecon_pytorch/nvdiffrast
pip install .
  1. Install Arcface Pytorch:
cd ..    # ./Deep3DFaceRecon_pytorch
git clone https://github.com/deepinsight/insightface.git
cp -r ./insightface/recognition/arcface_torch/ ./models/

Inference with a pre-trained model

Prepare prerequisite models

  1. Our method uses Basel Face Model 2009 (BFM09) to represent 3d faces. Get access to BFM09 using this link. After getting the access, download "01_MorphableModel.mat". In addition, we use an Expression Basis provided by Guo et al.. Download the Expression Basis (Exp_Pca.bin) using this link (google drive). Organize all files into the following structure:
Deep3DFaceRecon_pytorch
│
└─── BFM
    │
    └─── 01_MorphableModel.mat
    │
    └─── Exp_Pca.bin
    |
    └─── ...
  1. We provide a model trained on a combination of CelebA, LFW, 300WLP, IJB-A, LS3D-W, and FFHQ datasets. Download the pre-trained model using this link (google drive) and organize the directory into the following structure:
Deep3DFaceRecon_pytorch
│
└─── checkpoints
    │
    └─── <model_name>
        │
        └─── epoch_20.pth

Test with custom images

To reconstruct 3d faces from test images, organize the test image folder as follows:

Deep3DFaceRecon_pytorch
│
└─── <folder_to_test_images>
    │
    └─── *.jpg/*.png
    |
    └─── detections
        |
	└─── *.txt

The *.jpg/*.png files are test images. The *.txt files are detected 5 facial landmarks with a shape of 5x2, and have the same name as the corresponding images. Check ./datasets/examples for a reference.

Then, run the test script:

# get reconstruction results of your custom images
python test.py --name=<model_name> --epoch=20 --img_folder=<folder_to_test_images>

# get reconstruction results of example images
python test.py --name=<model_name> --epoch=20 --img_folder=./datasets/examples

Results will be saved into ./checkpoints/<model_name>/results/<folder_to_test_images>, which contain the following files:

*.png A combination of cropped input image, reconstructed image, and visualization of projected landmarks.
*.obj Reconstructed 3d face mesh with predicted color (texture+illumination) in the world coordinate space. Best viewed in Meshlab.
*.mat Predicted 257-dimensional coefficients and 68 projected 2d facial landmarks. Best viewd in Matlab.

Training a model from scratch

Prepare prerequisite models

  1. We rely on Arcface to extract identity features for loss computation. Download the pre-trained model from Arcface using this link. By default, we use the resnet50 backbone (ms1mv3_arcface_r50_fp16), organize the download files into the following structure:
Deep3DFaceRecon_pytorch
│
└─── checkpoints
    │
    └─── recog_model
        │
        └─── ms1mv3_arcface_r50_fp16
	    |
	    └─── backbone.pth
  1. We initialize R-Net using the weights trained on ImageNet. Download the weights provided by PyTorch using this link, and organize the file as the following structure:
Deep3DFaceRecon_pytorch
│
└─── checkpoints
    │
    └─── init_model
        │
        └─── resnet50-0676ba61.pth
  1. We provide a landmark detector (tensorflow model) to extract 68 facial landmarks for loss computation. The detector is trained on 300WLP, LFW, and LS3D-W datasets. Download the trained model using this link (google drive) and organize the file as follows:
Deep3DFaceRecon_pytorch
│
└─── checkpoints
    │
    └─── lm_model
        │
        └─── 68lm_detector.pb

Data preparation

  1. To train a model with custom images,5 facial landmarks of each image are needed in advance for an image pre-alignment process. We recommend using dlib or MTCNN to detect these landmarks. Then, organize all files into the following structure:
Deep3DFaceRecon_pytorch
│
└─── datasets
    │
    └─── <folder_to_training_images>
        │
        └─── *.png/*.jpg
	|
	└─── detections
            |
	    └─── *.txt

The *.txt files contain 5 facial landmarks with a shape of 5x2, and should have the same name with their corresponding images.

  1. Generate 68 landmarks and skin attention mask for images using the following script:
# preprocess training images
python data_preparation.py --img_folder <folder_to_training_images>

# alternatively, you can preprocess multiple image folders simultaneously
python data_preparation.py --img_folder <folder_to_training_images1> <folder_to_training_images2> <folder_to_training_images3>

# preprocess validation images
python data_preparation.py --img_folder <folder_to_validation_images> --mode=val

The script will generate files of landmarks and skin masks, and save them into ./datasets/<folder_to_training_images>. In addition, it also generates a file containing the path of all training data into ./datalist which will then be used in the training script.

Train the face reconstruction network

Run the following script to train a face reconstruction model using the pre-processed data:

# train with single GPU
python train.py --name=<custom_experiment_name> --gpu_ids=0

# train with multiple GPUs
python train.py --name=<custom_experiment_name> --gpu_ids=0,1

# train with other custom settings
python train.py --name=<custom_experiment_name> --gpu_ids=0 --batch_size=32 --n_epochs=20

Training logs and model parameters will be saved into ./checkpoints/<custom_experiment_name>.

By default, the script uses a batchsize of 32 and will train the model with 20 epochs. For reference, the pre-trained model in this repo is trained with the default setting on a image collection of 300k images. A single iteration takes 0.8~0.9s on a single Tesla M40 GPU. The total training process takes around two days.

To use a trained model, see Inference section.

Contact

If you have any questions, please contact the paper authors.

Citation

Please cite the following paper if this model helps your research:

@inproceedings{deng2019accurate,
    title={Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set},
    author={Yu Deng and Jiaolong Yang and Sicheng Xu and Dong Chen and Yunde Jia and Xin Tong},
    booktitle={IEEE Computer Vision and Pattern Recognition Workshops},
    year={2019}
}

The face images on this page are from the public CelebA dataset released by MMLab, CUHK.

Part of the code in this implementation takes CUT as a reference.

Comments
  • A test error in Ubuntu 18.04: ninja: build stopped: subcommand failed.

    A test error in Ubuntu 18.04: ninja: build stopped: subcommand failed.

    Hi, it was very excellent work. I meet some errors when I test the examples on the Ubuntu18.08 or Ubuntu 20.04. In Ubuntu18.04 the torch was 1.6 and the torch version was 1.8 in Ubunbu20.04. Here is the information in the terminal: Thanks for your attention.

    (deep3d_pytorch) heizhima@heizhima-Super-Server:~/PycharmProjects/WXL/Deep3DFaceRecon_pytorch$ python test.py --name=pretrained --epoch=20 --img_folder=./datasets/examples/
    /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/kornia/augmentation/augmentation.py:1875: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead.
      category=DeprecationWarning,
    /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
      import imp
    ----------------- Options ---------------
                    add_image: True                          
                   bfm_folder: BFM                           
                    bfm_model: BFM_model_front.mat           
                     camera_d: 10.0                          
                       center: 112.0                         
              checkpoints_dir: ./checkpoints                 
                 dataset_mode: None                          
                     ddp_port: 12355                         
            display_per_batch: True                          
                        epoch: 20                                   [default: latest]
              eval_batch_nums: inf                           
                        focal: 1015.0                        
                      gpu_ids: 0                             
                   img_folder: ./datasets/examples/                 [default: examples]
                    init_path: checkpoints/init_model/resnet50-0676ba61.pth
                      isTrain: False                                [default: None]
                        model: facerecon                     
                         name: pretrained                           [default: face_recon]
                    net_recon: resnet50                      
                        phase: test                          
                       suffix:                               
                      use_ddp: False                                [default: True]
                  use_last_fc: False                         
                      verbose: False                         
               vis_batch_nums: 1                             
                   world_size: 1                             
                        z_far: 15.0                          
                       z_near: 5.0                           
    ----------------- End -------------------
    model [FaceReconModel] was created
    loading the model from ./checkpoints/pretrained/epoch_20.pth
    0 ./datasets/examples/000002.jpg
    Traceback (most recent call last):
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1515, in _run_ninja_build
        env=env)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/subprocess.py", line 438, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "test.py", line 72, in <module>
        main(0, opt,opt.img_folder)
      File "test.py", line 62, in main
        model.test()           # run inference
      File "/home/heizhima/PycharmProjects/WXL/Deep3DFaceRecon_pytorch/models/base_model.py", line 162, in test
        self.forward()
      File "/home/heizhima/PycharmProjects/WXL/Deep3DFaceRecon_pytorch/models/facerecon_model.py", line 139, in forward
        self.pred_vertex, self.facemodel.face_buf, feat=self.pred_color)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
        result = self.forward(*input, **kwargs)
      File "/home/heizhima/PycharmProjects/WXL/Deep3DFaceRecon_pytorch/util/nvdiffrast.py", line 57, in forward
        self.glctx = dr.RasterizeGLContext(device=device)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/ops.py", line 151, in __init__
        self.cpp_wrapper = _get_plugin().RasterizeGLStateWrapper(output_db, mode == 'automatic', cuda_device_idx)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/ops.py", line 84, in _get_plugin
        torch.utils.cpp_extension.load(name=plugin_name, sources=source_paths, extra_cflags=opts, extra_cuda_cflags=opts, extra_ldflags=ldflags, with_cuda=True, verbose=False)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 974, in load
        keep_intermediates=keep_intermediates)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1179, in _jit_compile
        with_cuda=with_cuda)
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1279, in _write_ninja_file_and_build_library
        error_prefix="Error building extension '{}'".format(name))
      File "/home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
        raise RuntimeError(message)
    RuntimeError: Error building extension 'nvdiffrast_plugin': [1/4] c++ -MMD -MF torch_rasterize.o.d -DTORCH_EXTENSION_NAME=nvdiffrast_plugin -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /usr/local/cuda-10.2/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -DNVDR_TORCH -c /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/torch_rasterize.cpp -o torch_rasterize.o 
    FAILED: torch_rasterize.o 
    c++ -MMD -MF torch_rasterize.o.d -DTORCH_EXTENSION_NAME=nvdiffrast_plugin -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /usr/local/cuda-10.2/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -DNVDR_TORCH -c /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/torch_rasterize.cpp -o torch_rasterize.o 
    In file included from /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/../common/rasterize.h:42:0,
                     from /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/torch_rasterize.cpp:12:
    /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/torch/../common/glutil.h:36:21: fatal error: EGL/egl.h: No such file or directory
    compilation terminated.
    [2/4] c++ -MMD -MF rasterize.o.d -DTORCH_EXTENSION_NAME=nvdiffrast_plugin -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /usr/local/cuda-10.2/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -DNVDR_TORCH -c /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/rasterize.cpp -o rasterize.o 
    FAILED: rasterize.o 
    c++ -MMD -MF rasterize.o.d -DTORCH_EXTENSION_NAME=nvdiffrast_plugin -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /usr/local/cuda-10.2/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -DNVDR_TORCH -c /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/rasterize.cpp -o rasterize.o 
    In file included from /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/rasterize.h:42:0,
                     from /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/rasterize.cpp:9:
    /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/glutil.h:36:21: fatal error: EGL/egl.h: No such file or directory
    compilation terminated.
    [3/4] c++ -MMD -MF glutil.o.d -DTORCH_EXTENSION_NAME=nvdiffrast_plugin -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /usr/local/cuda-10.2/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -DNVDR_TORCH -c /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/glutil.cpp -o glutil.o 
    FAILED: glutil.o 
    c++ -MMD -MF glutil.o.d -DTORCH_EXTENSION_NAME=nvdiffrast_plugin -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/TH -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/include/THC -isystem /usr/local/cuda-10.2/include -isystem /home/heizhima/anaconda3/envs/deep3d_pytorch/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -DNVDR_TORCH -c /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/glutil.cpp -o glutil.o 
    In file included from /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/glutil.cpp:14:0:
    /home/heizhima/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/nvdiffrast/common/glutil.h:36:21: fatal error: EGL/egl.h: No such file or directory
    compilation terminated.
    ninja: build stopped: subcommand failed.
    
    opened by switch626 25
  • [F glutil.cpp:338] eglInitialize() failed Aborted (core dumped)

    [F glutil.cpp:338] eglInitialize() failed Aborted (core dumped)

    Greate work. I'm having a problem on running the inference code. Firstly, I meeted the problem of fatal error: EGL/egl.h: No such file or directory. I installed a package by apt-get install libegl1-mesa-dev. Then I met another problem [F glutil.cpp:338] eglInitialize() failed Aborted (core dumped)., Do you have any ideas how to fix this problem? 企业微信截图_1626788788701 企业微信截图_16267887991412 企业微信截图_1626788741329

    My Linux Environment: cudnn: 7.6 cuda: 10.2 python :3.6 pytorch:1.6.0

    good first issue 
    opened by yuzhou164 21
  • No output file

    No output file

    (deep3d_pytorch) PS D:\Deep3DFaceRecon_pytorch> python test.py --name=CelebA --epoch=20 --img_folder=./datasets/examples D:\anaconda3\envs\deep3d_pytorch\lib\site-packages\kornia\augmentation\augmentation.py:1875: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead. category=DeprecationWarning, D:\anaconda3\envs\deep3d_pytorch\lib\site-packages\torch\utils\cpp_extension.py:4: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp ----------------- Options --------------- add_image: True bfm_folder: BFM bfm_model: BFM_model_front.mat camera_d: 10.0 center: 112.0 checkpoints_dir: ./checkpoints dataset_mode: None ddp_port: 12355 display_per_batch: True epoch: 20 [default: latest] eval_batch_nums: inf focal: 1015.0 gpu_ids: 0 img_folder: ./datasets/examples [default: examples] init_path: checkpoints/init_model/resnet50-0676ba61.pth isTrain: False [default: None] model: facerecon name: CelebA [default: face_recon] net_recon: resnet50 phase: test suffix: use_ddp: False [default: True] use_last_fc: False verbose: False vis_batch_nums: 1 world_size: 1 z_far: 15.0 z_near: 5.0 ----------------- End ------------------- model [FaceReconModel] was created loading the model from ./checkpoints\CelebA\epoch_20.pth 0 ./datasets/examples\000002.jpg 1 ./datasets/examples\000006.jpg 2 ./datasets/examples\000007.jpg 3 ./datasets/examples\000031.jpg 4 ./datasets/examples\000033.jpg 5 ./datasets/examples\000037.jpg 6 ./datasets/examples\000050.jpg 7 ./datasets/examples\000055.jpg 8 ./datasets/examples\000114.jpg 9 ./datasets/examples\000125.jpg 10 ./datasets/examples\000126.jpg 11 ./datasets/examples\015259.jpg 12 ./datasets/examples\015270.jpg 13 ./datasets/examples\015309.jpg 14 ./datasets/examples\015310.jpg 15 ./datasets/examples\015316.jpg 16 ./datasets/examples\015384.jpg 17 ./datasets/examples\vd006.png 18 ./datasets/examples\vd025.png 19 ./datasets/examples\vd026.png 20 ./datasets/examples\vd034.png 21 ./datasets/examples\vd051.png 22 ./datasets/examples\vd070.png 23 ./datasets/examples\vd092.png 24 ./datasets/examples\vd102.png

    opened by Michaelwhite34 14
  • Outputs nothing in any folder, no clue how to fix.

    Outputs nothing in any folder, no clue how to fix.

    /home/a/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/kornia/augmentation/augmentation.py:1875: DeprecationWarning: GaussianBlur is no longer maintained and will be removed from the future versions. Please use RandomGaussianBlur instead.
      category=DeprecationWarning,
    ----------------- Options ---------------
                    add_image: True
                   bfm_folder: BFM
                    bfm_model: BFM_model_front.mat
                     camera_d: 10.0
                       center: 112.0
              checkpoints_dir: ./checkpoints
                 dataset_mode: None
                     ddp_port: 12355
            display_per_batch: True
                        epoch: 20                                   [default: latest]
              eval_batch_nums: inf
                        focal: 1015.0
                      gpu_ids: 0
                   img_folder: ./me                                 [default: examples]
                    init_path: checkpoints/init_model/resnet50-0676ba61.pth
                      isTrain: False                                [default: None]
                        model: facerecon
                         name: a                                    [default: face_recon]
                    net_recon: resnet50
                        phase: test
                       suffix:
                      use_ddp: False                                [default: True]
                  use_last_fc: False
                      verbose: False
               vis_batch_nums: 1
                   world_size: 1
                        z_far: 15.0
                       z_near: 5.0
    ----------------- End -------------------
    /home/a/anaconda3/envs/deep3d_pytorch/lib/python3.6/site-packages/torch/cuda/__init__.py:125: UserWarning:
    NVIDIA GeForce RTX 3080 with CUDA capability sm_86 is not compatible with the current PyTorch installation.
    The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 compute_37.
    If you want to use the NVIDIA GeForce RTX 3080 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/```
    
      warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
    model [FaceReconModel] was created
    loading the model from ./checkpoints/a/epoch_20.pth
    0 ./me/IMG20220728104454 - Copy.jpg
    1 ./me/IMG20220728104454.jpg
    

    Nothing gets outputted in either the results folder or in the images folder.

    My command is python test.py --name=a --epoch=20 --img_folder=./me

    opened by GarnetSunset 12
  • pytorch3d instead of nvdiffrast

    pytorch3d instead of nvdiffrast

    The nvdiffrast renderer has a very restrictive license policy which prevents using this code for anything but research (unlike in the TF repo). Is there a plan to add a support for pytorch3d or any other alternative differentiable renderer with a more permissive license?

    opened by jonilaserson 11
  • Error C7528: OpenGL reserves names starting with 'gl_': gl_PrimitiveID

    Error C7528: OpenGL reserves names starting with 'gl_': gl_PrimitiveID

    Hi,

    I got the following exception trying to run:

    $ python test.py --name=model1 --epoch=20 --img_folder=./datasets/mydata

    This returns an error:

    Error C7528: OpenGL reserves names starting with 'gl_': gl_PrimitiveID Error C2003: Incompatible options for Link

    Has anyone else experienced this? I am running on Windows with NVIDIA GeForce 3090.

    Kind Regards.

    opened by geek0075 7
  • How to get 3D keypoints in order to evaluate in NoW challenge?

    How to get 3D keypoints in order to evaluate in NoW challenge?

    Thanks for your greate work! I'm a recruit for 3d reconstruction , and I want to evaluate my model in NoW challenge. I follow the guide from https://github.com/soubhiksanyal/now_evaluation

    python check_predictions.py <predicted_mesh_path> <predicted_mesh_landmark_path> <gt_scan_path> <gt_lmk_path> 
    

    There is a little doubt about the <predicted_mesh_landmark_path>. It looks like the file containing 3D coordinates of 7 key points. Can you give me some advice or guidance on obtaining the 3D coordinates of key points? Thanks again!

    opened by zhih-li 7
  • 关于NoW challenge?

    关于NoW challenge?

    想问一下是如何得到NoW challenge里人脸图片的5 lmk,是人脸图片直接用"How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks"的3D lmk 检测还是用了NoW里提供的bbox直接裁剪成256x256?参加挑战的模型是公开的epoch_20.pth吗?生成的obj文件里顶点的精度是np.float32吗?

    opened by zyqlalala 7
  • how to get the *.txt file about facial landmarks?

    how to get the *.txt file about facial landmarks?

    In Readme.md, "The .jpg/.png files are test images. *The .txt files are detected 5 facial landmarks with a shape of 5x2, and have the same name as the corresponding images. Check ./datasets/examples for a reference." I want to know how to get them.

    opened by sablea 5
  • Loss Weights in repository different from those in your paper

    Loss Weights in repository different from those in your paper

    Dear Authors,

    Thanks for sharing your code! I found in this repository the w_feat is set to 1.0, w_id=0.2, and w_tex=1.7e-2 while in your paper they are set to w_per=0.2, w_alpha=1.0 and w_gamma=1.7e-3 (this w_gamma is the w_tex in this repository). Could you please tell me why they are different from what's in your paper? Besides I found neither the settings in your paper nor the settings in this repository can give me a model as good as what you released. Could you please help me to find what's wrong? I used MTCNN to extract the 5 facial landmarks and followed your data preparation procedure to prepare the training data. For the images in 300W-LP and LS3D-W datasets, I loaded the ground truth 68 landmarks directly and used the method extract_5p() in this repository to extract the 5 facial landmarks. I didn't balance the pose and race distributions, will this influence much? Thank you very much for reading this issue!

    opened by humannumber196 5
  • Multi images inferencing

    Multi images inferencing

    Hi @YuDeng , Thank you for sharing your great work.

    How can I do model inferencing for the multi-image case as there is no C-Net available yet? My dataset has three images for each person and was taken from frontal and side.

    You mentioned we could use the average shape for the multi-image case in #17 . I'm new to this area. Could you please expand on this?

    If you could provide some example code would be much appreciated.

    Thanks

    opened by icedpanda 5
  • May I know how to solve this problem or is it cause by the Cuda version or library?

    May I know how to solve this problem or is it cause by the Cuda version or library?

    File "/home/ling/Desktop/Deep3DFaceRecon_pytorch/util/preprocess.py", line 195, in align_img trans_params = np.array([w0, h0, s, t[0], t[1]]) ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (5,) + inhomogeneous part.

    opened by lingleong981130 0
  • How to choose 300k images from these datasets?

    How to choose 300k images from these datasets?

    Hi, thanks for your excellent work. I collected the 6 face datasets you mentioned and attempted to train the model by myself, but result was worse than yours. can you provide your datalist and let me know how to choose 300k images?

    opened by hrg0417 0
  • Is it possible to change the expression of a face in the generated model?

    Is it possible to change the expression of a face in the generated model?

    For example, if the photo has a smiling expression, I would like it to generate a standard expression without smiling, is this possible? Or is there another repository that can help me achieve this?

    opened by agchaowanhui 1
  • What does each dimension of expbase represent to which part of face?

    What does each dimension of expbase represent to which part of face?

    Hi,

    Thanks for the amazing work. I am wondering what does each dimension of expbase represent in this file at L52. According to it there are 64 dimensions in expBase, I am wondering that what does each dimension for which part of face, is there any document for the description I can refer to? Many thanks.

    opened by keishatsai 0
Owner
Sicheng Xu
Sicheng Xu
Weakly Supervised Dense Event Captioning in Videos, i.e. generating multiple sentence descriptions for a video in a weakly-supervised manner.

WSDEC This is the official repo for our NeurIPS paper Weakly Supervised Dense Event Captioning in Videos. Description Repo directories ./: global conf

Melon(Xuguang Duan) 96 Nov 1, 2022
PyTorch version of the paper 'Enhanced Deep Residual Networks for Single Image Super-Resolution' (CVPRW 2017)

About PyTorch 1.2.0 Now the master branch supports PyTorch 1.2.0 by default. Due to the serious version problem (especially torch.utils.data.dataloade

Sanghyun Son 2.1k Jan 1, 2023
Script that receives an Image (original) and a set of images to be used as "pixels" in reconstruction of the Original image using the set of images as "pixels"

picinpics Script that receives an Image (original) and a set of images to be used as "pixels" in reconstruction of the Original image using the set of

RodrigoCMoraes 1 Oct 24, 2021
Face Library is an open source package for accurate and real-time face detection and recognition

Face Library Face Library is an open source package for accurate and real-time face detection and recognition. The package is built over OpenCV and us

null 52 Nov 9, 2022
PyTorch implementation of the Deep SLDA method from our CVPRW-2020 paper "Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis"

Lifelong Machine Learning with Deep Streaming Linear Discriminant Analysis This is a PyTorch implementation of the Deep Streaming Linear Discriminant

Tyler Hayes 41 Dec 25, 2022
Code for Dual Contrastive Learning for Unsupervised Image-to-Image Translation, NTIRE, CVPRW 2021.

arXiv Dual Contrastive Learning Adversarial Generative Networks (DCLGAN) We provide our PyTorch implementation of DCLGAN, which is a simple yet powerf

null 119 Dec 4, 2022
[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

SADRNet Paper link: SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction Requirements python

Multimedia Computing Group, Nanjing University 99 Dec 30, 2022
Code for weakly supervised segmentation of a single class

SingleClassRL Implementation of weak single object segmentation from paper "Regularized Loss for Weakly Supervised Single Class Semantic Segmentation"

null 16 Nov 14, 2022
Official PyTorch implementation of "IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos", CVPRW 2021

IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos Introduction This repo is official PyTorch implementatio

Gyeongsik Moon 29 Sep 24, 2022
This repo is official PyTorch implementation of MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices(CVPRW 2021).

Github Code of "MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices" Introduction This repo is official PyTorch implementatio

Choi Sang Bum 203 Jan 5, 2023
Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal Action Localization' (ICCV-21 Oral)

Learning-Action-Completeness-from-Points Official Pytorch Implementation of 'Learning Action Completeness from Points for Weakly-supervised Temporal A

Pilhyeon Lee 67 Jan 3, 2023
Official PyTorch implementation of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image", ICCV 2019

PoseNet of "Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image" Introduction This repo is official Py

Gyeongsik Moon 677 Dec 25, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 5 Dec 10, 2022
Automatically download the cwru data set, and then divide it into training data set and test data set

Automatically download the cwru data set, and then divide it into training data set and test data set.自动下载cwru数据集,然后分训练数据集和测试数据集

null 6 Jun 27, 2022
Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR 2022)

Voxel Set Transformer: A Set-to-Set Approach to 3D Object Detection from Point Clouds (CVPR2022)[paper] Authors: Chenhang He, Ruihuang Li, Shuai Li, L

Billy HE 141 Dec 30, 2022
Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency(ECCV 2020) This is an official python implementati

null 304 Jan 3, 2023
A Fast and Accurate One-Stage Approach to Visual Grounding, ICCV 2019 (Oral)

One-Stage Visual Grounding ***** New: Our recent work on One-stage VG is available at ReSC.***** A Fast and Accurate One-Stage Approach to Visual Grou

Zhengyuan Yang 118 Dec 5, 2022
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Felix Wimbauer 494 Jan 6, 2023