Code for "Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency" paper

Last update: Jan 6, 2023

Overview

UNICORN 🦄

Webpage | Paper | BibTex

PyTorch implementation of "Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency" paper, check out our webpage for details!

If you find this code useful, don't forget to star the repo ⭐ and cite the paper:

@article{monnier2022unicorn,
  title={{Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance 
  Consistency}},
  author={Monnier, Tom and Fisher, Matthew and Efros, Alexei A and Aubry, Mathieu},
  journal={arXiv:2204.10310 [cs]},
  year={2022},
}

Installation 👷

1. Create conda environment 🔧

conda env create -f environment.yml
conda activate unicorn

Optional: some monitoring routines are implemented, you can use them by specifying your visdom port in the config file. You will need to install visdom from source beforehand

git clone https://github.com/facebookresearch/visdom
cd visdom && pip install -e .

2. Download datasets ⬇️

bash scripts/download_data.sh

This command will download one of the following datasets:

ShapeNet NMR: paper / NMR paper / dataset (33Go, thanks to the DVR team for hosting the data)
CUB-200: paper / webpage / dataset (1Go)
Pascal3D+ Cars: paper / webpage (including ftp download link, 7.5Go) / UCMR annotations (thanks to the UCMR team for releasing them)
CompCars: paper / webpage / dataset (12Go, thanks to the GIRAFFE team for hosting the data)
LSUN: paper / webpage / horse dataset (69Go) / moto dataset (42Go)

3. Download pretrained models ⬇️

bash scripts/download_model.sh

This command will download one of the following models:

car.pkl trained on CompCars: gdrive link
bird.pkl trained on CUB-200: gdrive link
moto.pkl trained on LSUN Motorbike: gdrive link
horse.pkl trained on LSUN Horse: gdrive link
sn_*.pkl trained on each ShapeNet category: airplane, bench, cabinet, car, chair, display, lamp, phone, rifle, sofa, speaker, table, vessel

NB: it may happen that gdown hangs, if so you can download them manually with the gdrive links and move them to the models folder.

How to use 🚀

1. 3D reconstruction of car images 🚘

You first need to download the car model (see above), then launch:

cuda=gpu_id model=car.pkl input=demo ./scripts/reconstruct.sh

where:

gpu_id is a target cuda device id,
car.pkl corresponds to a pretrained model,
demo is a folder containing the target images.

It will create a folder demo_rec containing the reconstructed meshes (.obj format + gif visualizations).

2. Reproduce our results 📊

To launch a training from scratch, run:

cuda=gpu_id config=filename.yml tag=run_tag ./scripts/pipeline.sh

where:

gpu_id is a target cuda device id,
filename.yml is a YAML config located in configs folder,
run_tag is a tag for the experiment.

Results are saved at runs/${DATASET}/${DATE}_${run_tag} where DATASET is the dataset name specified in filename.yml and DATE is the current date in mmdd format. Some training visual results like reconstruction examples will be saved. Available configs are:

sn/*.yml for each ShapeNet category
car.yml for CompCars dataset
cub.yml for CUB-200 dataset
horse.yml for LSUN Horse dataset
moto.yml for LSUN Motorbike dataset
p3d_car.yml for Pascal3D+ Car dataset

3. Train on a custom dataset 🔮

If you want to learn a model for a custom object category, here are the key things you need to do:

put your images in a custom_name folder inside the datasets folder
write a config custom.yml with custom_name as dataset.name and move it to the configs folder: as a rule of thumb for the progressive conditioning milestones, put the number of epochs corresponding to 500k iterations for each stage
launch training with:

cuda=gpu_id config=custom.yml tag=custom_run_tag ./scripts/pipeline.sh

Further information 📚

If you like this project, check out related works from our group:

Code for the paper Learning the Predictability of the Future

Learning the Predictability of the Future Code from the paper Learning the Predictability of the Future. Website of the project in hyperfuture.cs.colu

Computer Vision Lab at Columbia University

139 Nov 18, 2022

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning This is the PyTorch implementation of our paper: FeatMatch: Feature-Based Augmentat

43 Nov 19, 2022

Code for the paper A Theoretical Analysis of the Repetition Problem in Text Generation

A Theoretical Analysis of the Repetition Problem in Text Generation This repository share the code for the paper "A Theoretical Analysis of the Repeti

37 Nov 21, 2022

Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks (paper) By Qing-Long Zhang and Yu-Bin Yang [State Key Laboratory for Novel Software T

199 Jan 8, 2023

Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Non-Rigid Neural Radiance Fields This is the official repository for the project "Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synt

296 Dec 29, 2022

Comments

The result of your model you provided is a little wrong with the images download from the website

I am very appreciated with your work. I download the images from the website.I found the car, bird, and horse model can not work well with the images.I can get the shape and the texture in the result, but the texture, even the colour is wrong. And the motobike model works well.

opened by lisentao 6
.yml file to train from scratch

Hi, your work is impressive and much appreciated.

I'm trying to train a model from scratch but, even though in the README appears a tutorial to do so, i think there are no examples of .yml files in which training from scratch is specified so i'm a bit lost. What do i have to include in the .yml to train from scratch?

opened by Mohgwyn 3
FileNotFoundError: [Errno 2] No such file or directory:

hi , I'm new in AI I'm training the model from scratch i download the dataset and the package

I run in google colab: !cuda=GPU-f491dbb6-2dc7-a094-5417-a57de28df194 config=p3d_car.yml tag=run_tag /content/unicorn-main\scripts\pipeline.sh

this appears to me:

Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/scipy/io/matlab/mio.py", line 39, in _open_file return open(file_like, mode), True FileNotFoundError: [Errno 2] No such file or directory: '/content/unicorn-main/datasets/pascal_3d/ucmr_anno/data/car_train.mat'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/content/unicorn-main/src/trainer.py", line 292, in trainer = Trainer(cfg, run_dir, seed=seed) File "/content/unicorn-main/src/utils/init.py", line 104, in wrapper return f(*args, **kw) File "/content/unicorn-main/src/trainer.py", line 41, in init self.train_loader, self.val_loader, self.test_loader = create_train_val_test_loader(cfg, rank, world_size) File "/content/unicorn-main/src/dataset/init.py", line 16, in create_train_val_test_loader train = get_dataset(name)(split="train", **kwargs) File "/content/unicorn-main/src/dataset/p3d_car.py", line 38, in init self.data = scio.loadmat(str(path), struct_as_record=False, squeeze_me=True)['images'] File "/usr/local/lib/python3.7/dist-packages/scipy/io/matlab/mio.py", line 224, in loadmat with _open_file_context(file_name, appendmat) as f: File "/usr/lib/python3.7/contextlib.py", line 112, in enter return next(self.gen) File "/usr/local/lib/python3.7/dist-packages/scipy/io/matlab/mio.py", line 17, in _open_file_context f, opened = _open_file(file_like, appendmat, mode) File "/usr/local/lib/python3.7/dist-packages/scipy/io/matlab/mio.py", line 45, in _open_file return open(file_like, mode), True FileNotFoundError: [Errno 2] No such file or directory: '/content/unicorn-main/datasets/pascal_3d/ucmr_anno/data/car_train.mat'

i just want to know where the location of ucmr_anno file install so i can fix it and know the path

i spend so much time searching ucmr_anno file in colab files

help me please and thank for the work

opened by Jawadllo 2

Code for "Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency" paper

Related tags

Overview

UNICORN 🦄

Webpage | Paper | BibTex

Installation 👷

1. Create conda environment 🔧

2. Download datasets ⬇️

3. Download pretrained models ⬇️

How to use 🚀

1. 3D reconstruction of car images 🚘

2. Reproduce our results 📊

3. Train on a custom dataset 🔮

Further information 📚

You might also like...

Code for the paper Learning the Predictability of the Future

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

Code for the paper A Theoretical Analysis of the Repetition Problem in Text Generation

Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

Open source repository for the code accompanying the paper 'Non-Rigid Neural Radiance Fields Reconstruction and Novel View Synthesis of a Deforming Scene from Monocular Video'.

Code for the Shortformer model, from the paper by Ofir Press, Noah A. Smith and Mike Lewis.

PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

Official code for paper "Optimization for Oriented Object Detection via Representation Invariance Loss".

Code for our CVPR 2021 paper "MetaCam+DSCE"

Comments

The result of your model you provided is a little wrong with the images download from the website

.yml file to train from scratch

FileNotFoundError: [Errno 2] No such file or directory:

Owner

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Code for ACM MM 2020 paper "NOH-NMS: Improving Pedestrian Detection by Nearby Objects Hallucination"

Official TensorFlow code for the forthcoming paper

This is the code for the paper "Contrastive Clustering" (AAAI 2021)