Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Related tags

Deep Learning WLDO
Overview

Who Left the Dogs Out?

Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Disclaimer

Please note, this repository is in beta while I make bug fixes etc.

Install

Clone the repository with submodules:

git clone --recurse-submodules https://github.com/benjiebob/WLDO

For segmentation decoding, install pycocotools python -m pip install "git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI"

Datasets

To use the StanfordExtra dataset, you will need to download the .json file via the repository.

Please ensure you have StanfordExtra_v12 installed, which we released 1 Feb 2021.

You may also wish to evaluate the Animal Pose Dataset. If so, download all of the dog images into data/animal_pose/images. For example, an image path should look like: data/animal_pose/images/2007_000063.jpg. We have reformatted the annotation file and enclose it in this repository data/animal_pose/animal_pose_data.json.

Splits

The train/validation/test splits used for our ECCV 2020 submission are contained in the data/StanfordExtra_v12 repository and under the data/animal_pose folder.

Pretrained model

Please download our pretrained model and place underneath data/pretrained/3501_00034_betas_v4.pth.

Quickstart

Eval

To evaluate the performance of the model on the StanfordExtra dataset, run eval.py:

cd wldo_regressor
python eval.py --dataset stanford

You can also run on the animal_pose dataset

python eval.py --dataset animal_pose

Results

Dataset IOU PCK @ 0.15
Avg Legs Tail Ears Face
StanfordExtra 74.2 78.8 76.4 63.9 78.1 92.1
Animal Pose 67.5 67.6 60.4 62.7 86.0 86.7

Note that we have recently updated the tables in the arxiv version of our paper to account for some fixed dataset annotations and to use an improved version of the PCK metric. More details can be found in the paper.

Demo

To run the model on a series of images, place the images in a directory, and call the script demo.py. To see an example of this working, run demo.py and it will use the images in example_imgs:

cd wldo_regressor
python demo.py

Related Work

This repository owes a great deal to the following works and authors:

  • SMALify; Biggs et al. provided an energy minimization framework for fitting to animal video/images. A version of this was used as a baseline in this paper.
  • SMAL; Zuffi et al. designed the SMAL deformable quadruped template model and have provided me with wonderful advice/guidance throughout my PhD journey.
  • SMALST; Zuffi et al. provided PyTorch implementations of the SMAL skinning functions which have been used here.
  • SMPLify; Bogo et al. provided the basis for our original ChumPY implementation.

Acknowledgements

If you make use of this code, please cite the following paper:

@inproceedings{biggs2020wldo,
  title={{W}ho left the dogs out?: {3D} animal reconstruction with expectation maximization in the loop},
  author={Biggs, Benjamin and Boyne, Oliver and Charles, James and Fitzgibbon, Andrew and Cipolla, Roberto},
  booktitle={ECCV},
  year={2020}
}

Contribute

Please create a pull request or submit an issue if you would like to contribute.

Licensing

(c) Benjamin Biggs, Oliver Boyne, Andrew Fitzgibbon and Roberto Cipolla. Department of Engineering, University of Cambridge 2020

By downloading this dataset, you agree to the Creative Commons Attribution-NonCommercial 4.0 International license. This license allows users to use, share and adapt the dataset, so long as credit is given to the authors (e.g. by citation) and the dataset is not used for any commercial purposes.

THIS SOFTWARE AND ANNOTATIONS ARE PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

You might also like...
Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

Progressive Transformers for End-to-End Sign Language Production Source code for "Progressive Transformers for End-to-End Sign Language Production" (B

Dogs classification with Deep Metric Learning using some popular losses
Dogs classification with Deep Metric Learning using some popular losses

Tsinghua Dogs classification with Deep Metric Learning 1. Introduction Tsinghua Dogs dataset Tsinghua Dogs is a fine-grained classification dataset fo

ML model to classify between cats and dogs

Cats-and-dogs-classifier This is my first ML model which can classify between cats and dogs. Here the accuracy is around 75%, however , the accuracy c

PyTorch implementation of ECCV 2020 paper "Foley Music: Learning to Generate Music from Videos "

Foley Music: Learning to Generate Music from Videos This repo holds the code for the framework presented on ECCV 2020. Foley Music: Learning to Genera

[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.
[ECCV 2020] Reimplementation of 3DDFAv2, including face mesh, head pose, landmarks, and more.

Stable Head Pose Estimation and Landmark Regression via 3D Dense Face Reconstruction Reimplementation of (ECCV 2020) Towards Fast, Accurate and Stable

1st Place Solution to ECCV-TAO-2020: Detect and Represent Any Object for Tracking

Instead, two models for appearance modeling are included, together with the open-source BAGS model and the full set of code for inference. With this code, you can achieve around mAP@23 with TAO test set (based on our estimation).

Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)
Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020)

Causality In Traffic Accident (Under Construction) Repository for Traffic Accident Benchmark for Causality Recognition (ECCV 2020) Overview Data Prepa

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

Comments
  • Keypoint choosing.

    Keypoint choosing.

    Hi Benjamin, your WLDO is excellent, I applied it on other animals and it can estimate their pose very good. However, I have a question about the keypoint you choose. For the dogs, I only see one keypoint on the whole torso, the tail-start. If there is another point around the neck, then it seems reasonable. However, there are only few points on the head. How do this one point define the direction of the torso? Or the tail-start point and the four points (point 4,2,5,11) can define the whole body. Thank you.

    opened by Drow999 7
  • Neural renderer dependency

    Neural renderer dependency

    Good afternoon,

    May I ask how do you install the neural renderer dependency? If I run regular pip install, it just fails for me. I have tried to fix some issues there and install it manually by copying the source code, and install it from the setup.py. The WLDO code then seems to run, but results are pretty bad. If I compare with the image from the README, I can see that I have worse results (with the pre-trained network) then it probably should be.

    example

    I am not 100% sure, but I guess that it might be the problem with the neural renderer dependency. Which version did you use and how did you install it?

    Thank you.

    opened by iegorval 5
  • Operational demo.py problem

    Operational demo.py problem

    self.model_renderer = NeuralRenderer( config.IMG_RES, proj_type=config.PROJECTION, norm_f0=config.NORM_F0, norm_f=config.NORM_F, norm_z=config.NORM_Z) synth_rgb, synth_silhouettes = self.model_renderer( verts, faces, pred_camera) synth_rgb = torch.clamp(synth_rgb, 0.0, 1.0) TypeError: clamp(): argument 'input' (position 1) must be Tensor, not tuple

    Is there a problem with the torch version? Or is the library now incompatible?

    opened by nightmareisme 2
  • Bug in demo.py

    Bug in demo.py

    Issue reported by Le Jiang:

    There are two functions "load_model_from_disk()" and "Model()"which are used in both demo and eval. One has "shape_family_id" and "load_from_disk" variables and one doesn't. The load_model_from_disk is defined by yourself so it can be different, but the Model() is the function from neural_renderer I think and it needs shape_family_id and load_from_disk. And I cannot run it now. Can you use the demo or do you have any update on this? Thank you so much!

    opened by benjiebob 1
Owner
Benjamin Biggs
Benjamin Biggs
Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency(ECCV 2020) This is an official python implementati

null 304 Jan 3, 2023
Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

Nader Akoury 27 Dec 20, 2022
Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

null 54 Dec 15, 2022
Code for the paper: Adversarial Training Against Location-Optimized Adversarial Patches. ECCV-W 2020.

Adversarial Training Against Location-Optimized Adversarial Patches arXiv | Paper | Code | Video | Slides Code for the paper: Sukrut Rao, David Stutz,

Sukrut Rao 32 Dec 13, 2022
Code for the paper "Improving Vision-and-Language Navigation with Image-Text Pairs from the Web" (ECCV 2020)

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh

Arjun Majumdar 44 Dec 14, 2022
Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

Davis Rempe 207 Jan 5, 2023
PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

HAN PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network" This repository is for HAN introduced in the

五维空间 140 Nov 23, 2022
Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

Representation Robustness Evaluations Our implementation is based on code from MadryLab's robustness package and Devon Hjelm's Deep InfoMax. For all t

Sicheng 19 Dec 7, 2022
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

MultiModal-InfoMax This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Informa

Deep Cognition and Language Research (DeCLaRe) Lab 89 Dec 26, 2022
Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

Martin Li 85 Dec 22, 2022