KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

Related tags

Deep Learning keypoint_deformer
Overview

KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control

Tomas Jakab, Richard Tucker, Ameesh Makadia, Jiajun Wu, Noah Snavely, Angjoo Kanazawa. CVPR, 2021 (Oral presentation).

We present KeypointDeformer, a novel unsupervised method for shape control through automatically discovered 3D keypoints. Our approach produces intuitive and semantically consistent control of shape deformations. Moreover, our discovered 3D keypoints are consistent across object category instances despite large shape variations. Since our method is unsupervised, it can be readily deployed to new object categories without requiring expensive annotations for 3D keypoints and deformations.

Install

Clone the repo

git clone https://github.com/tomasjakab/keypoint_deformer
cd keypoint_deformer

Install using conda:

conda env create -f environment.yml 
conda activate keypointdeformer

Set-up python path:

export PYTHONPATH=$PYTHONPATH:$(pwd)

Training

Download ShapeNet to data/shapenet. The path to ShapeNet can be also customized in config files configs/* with the option mesh_dir.

To train a model on the airplane category with 8 unsupervised keypoints run:

python scripts/main.py -c configs/airplane-8kpt.yaml

To train a model on the chair category with 12 unsupervised keypoints run:

python scripts/main.py -c configs/chair-12kpt.yaml

Testing

To test the trained model run:

python scripts/main.py -c configs/airplane-8kpt.yaml -t configs/test.yaml

This will create result files in logs/airplane-8kpt/test/<SAMPLE NAME>. The file source_mesh.obj contains the input mesh and the file source_keypoints.txt predicted unsupervised keypoints.

To visualize the results run:

python browse3d/browse3d.py --log_dir logs/airplane-8kpt/test --port 5050

and open localhost:5050 in your web browser.

Demo

Try the interactive demo without any instalation.

Acknowledgments

Parts of the code are based on Neural Cages.

You might also like...
68 keypoint annotations for COFW test data

68 keypoint annotations for COFW test data This repository contains manually annotated 68 keypoints for COFW test data (original annotation of CFOW da

null 31 Dec 6, 2022
Solving SMPL/MANO parameters from keypoint coordinates.
Solving SMPL/MANO parameters from keypoint coordinates.

Minimal-IK A simple and naive inverse kinematics solver for MANO hand model, SMPL body model, and SMPL-H body+hand model. Briefly, given joint coordin

Yuxiao Zhou 305 Dec 30, 2022
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

UDP-Pose This is the pytorch implementation for UDP++, which won the Fisrt place in COCO Keypoint Challenge at ECCV 2020 Workshop. Top-Down Results on

null 20 Jul 29, 2022
ROS-UGV-Control-Interface - Control interface which can be used in any UGV
ROS-UGV-Control-Interface - Control interface which can be used in any UGV

ROS-UGV-Control-Interface Cam Closed: Cam Opened:

Ahmet Fatih Akcan 1 Nov 4, 2022
Hand Gesture Volume Control is AIML based project which uses image processing to control the volume of your Computer.
Hand Gesture Volume Control is AIML based project which uses image processing to control the volume of your Computer.

Hand Gesture Volume Control Modules There are basically three modules Handtracking Program Handtracking Module Volume Control Program Handtracking Pro

VITTAL 1 Jan 12, 2022
pytorch implementation of
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020
Seach Losses of our paper 'Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search', accepted by ICLR 2021.
Seach Losses of our paper 'Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search', accepted by ICLR 2021.

CSE-Autoloss Designing proper loss functions for vision tasks has been a long-standing research direction to advance the capability of existing models

Peidong Liu(刘沛东) 54 Dec 17, 2022
A PyTorch implementation of
A PyTorch implementation of "Pathfinder Discovery Networks for Neural Message Passing"

A PyTorch implementation of "Pathfinder Discovery Networks for Neural Message Passing" (WebConf 2021). Abstract In this work we propose Pathfind

Benedek Rozemberczki 49 Dec 1, 2022
Neighborhood Contrastive Learning for Novel Class Discovery

Neighborhood Contrastive Learning for Novel Class Discovery This repository contains the official implementation of our paper: Neighborhood Contrastiv

Zhun Zhong 56 Dec 9, 2022
Comments
  • Checkpoints?

    Checkpoints?

    Hi, I'm really interested in your work. I wonder if you have pre-trained checkpoints (like those used in your interactive demo), so that we can perform interactions on our own data.

    Thanks

    opened by evelinehong 0
  • Code about Categorical Shape Prior

    Code about Categorical Shape Prior

    Hello,

    Thanks for the code for this awesome work. I've got great interest in Categorical Shape Prior part in your paper. I wonder how to obtain the PCA basis of keypoints and how to "synchronize" keypoints based on such shape basis. Could you please release the code of this part?

    Thank you!

    opened by ZhijunGong 0
  • Interactive control

    Interactive control

    Hello,

    Thank you very much for sharing the code for this interesting work. I was able to train and generate test results using your code for the data you have used. I then used browse.py to visualize the results and have a few questions:

    1. browse.py seems to be showing the source object along with its keypoints (instead of the deformed object). But I was not able to interact with the keypoints like you have in your interactive demo. I tried toggling the "open controls" in the pop-up window but that did not work. Is there a way to enable interactive deformations?

    2. If we want to train or test your code with a completely different 3D mesh dataset (non-Shapenet), could you please provide some guidance on the steps we need to follow to prepare the dataset?

    Thank you.

    opened by athena913 0
Owner
Tomas Jakab
Tomas Jakab
GitHub
Implementation of "JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting"

JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting Pytorch implementation for the paper "JOKR: Joint Keypoint Repres

null 45 Dec 25, 2022
TraND: Transferable Neighborhood Discovery for Unsupervised Cross-domain Gait Recognition.

TraND This is the code for the paper "Jinkai Zheng, Xinchen Liu, Chenggang Yan, Jiyong Zhang, Wu Liu, Xiaoping Zhang and Tao Mei: TraND: Transferable

Jinkai Zheng 32 Apr 4, 2022
Unsupervised Discovery of Object Radiance Fields

Unsupervised Discovery of Object Radiance Fields by Hong-Xing Yu, Leonidas J. Guibas and Jiajun Wu from Stanford University. arXiv link: https://arxiv

Hong-Xing Yu 148 Nov 30, 2022
Pytorch implementation of the unsupervised object discovery method LOST.

LOST Pytorch implementation of the unsupervised object discovery method LOST. More details can be found in the paper: Localizing Objects with Self-Sup

Valeo.ai 189 Dec 25, 2022
[CVPR 2021] Unsupervised 3D Shape Completion through GAN Inversion

ShapeInversion Paper Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, Chen Change Loy "Unsupervised 3D

null 100 Dec 22, 2022
This repo is a PyTorch implementation for Paper "Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds"

Unsupervised Learning for Cuboid Shape Abstraction via Joint Segmentation from Point Clouds This repository is a PyTorch implementation for paper: Uns

Kaizhi Yang 42 Dec 9, 2022
OpenPose: Real-time multi-person keypoint detection library for body, face, hands, and foot estimation

Build Type Linux MacOS Windows Build Status OpenPose has represented the first real-time multi-person system to jointly detect human body, hand, facia

null 25.7k Jan 9, 2023
CharacterGAN: Few-Shot Keypoint Character Animation and Reposing

CharacterGAN Implementation of the paper "CharacterGAN: Few-Shot Keypoint Character Animation and Reposing" by Tobias Hinz, Matthew Fisher, Oliver Wan

Tobias Hinz 181 Dec 27, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Single-stage Keypoint-based Category-level Object Pose Estimation from an RGB Image

CenterPose Overview This repository is the official implementation of the paper "Single-stage Keypoint-based Category-level Object Pose Estimation fro

NVIDIA Research Projects 188 Dec 27, 2022