[NeurIPS 2021] Code for Learning Signal-Agnostic Manifolds of Neural Fields

Last update: Dec 12, 2022

Related tags

Text Data & NLP gem

Overview

Learning Signal-Agnostic Manifolds of Neural Fields

This is the uncleaned code for the paper Learning Signal-Agnostic Manifolds of Neural Fields. The cleaned code will be cleaned shortly.

Downloading Data

Please utilize the following link to download the underlying models and data used in the paper and extract it in the root directory. Please download the 3D shape dataset from here.

Demo

The underying audiovisual manifold illustrated in the paper may be constructed by utilizing the following command

python experiment_scripts/audiovisual_manifold_interpolate.py --experiment_name=audiovis_demo --checkpoint_path log_root/audiovis_demo/checkpoints/model_70000.pth

Training Different Signal Manifolds

Please utilize the following command to train an image manifold

python experiment_scripts/train_autodecoder_multiscale.py --experiment_name=celeba

Please utilize the following command to train a 3D shape manifold

python experiment_scripts/train_imnet_autodecoder.py --experiment_name=imnet

Please utilize the following command to train an audio manifold

python experiment_scripts/train_audio_autodecoder.py --experiment_name=audio

Please utilize the following command to train an audiovisual manifold

python experiment_scripts/train_audiovisual_autodecoder.py --experiment_name=audiovisual

Citing our Paper

If you find our code useful for your research, please consider citing

@inproceedings{du2021gem,
  title={Learing Signal-Agnostic Manifolds of Neural Fields},
  author={Du, Yilun and Collins, M. Katherine and and Tenenbaum, B. Joshua
  and Sitzmann, Vincent},
  booktitle={Advances in Neural Information Processing Systems},
  year={2021}
}

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering Paper: https://arxiv.org/abs/2103.00762 Running Run on the provided DTU scene cd run ba

68 Jan 6, 2023

NeurIPS'21: Probabilistic Margins for Instance Reweighting in Adversarial Training (Pytorch implementation).

source code for NeurIPS21 paper robabilistic Margins for Instance Reweighting in Adversarial Training

9 Dec 20, 2022

Code for CVPR 2021 paper: Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning

Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning This is the PyTorch companion code for the paper: A

69 Jan 3, 2023

Code for our paper "Transfer Learning for Sequence Generation: from Single-source to Multi-source" in ACL 2021.

TRICE: a task-agnostic transferring framework for multi-source sequence generation This is the source code of our work Transfer Learning for Sequence

9 Jun 27, 2022

Code for EMNLP 2021 main conference paper "Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification"

105 Jan 3, 2023

Code for Findings at EMNLP 2021 paper: "Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning"

Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning This repo is for Findings at EMNLP 2021 paper: Learn Cont

6 Sep 2, 2022

Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.

textgenrnn Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code, or quickly tr

4.8k Dec 30, 2022

Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.

textgenrnn Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code, or quickly tr

4.3k Feb 18, 2021

code for "AttentiveNAS Improving Neural Architecture Search via Attentive Sampling"

AttentiveNAS: Improving Neural Architecture Search via Attentive Sampling This repository contains PyTorch evaluation code, training code and pretrain

94 Oct 26, 2022

Comments

pretrained models

Hi @yilundu,

Congrats for the very interesting work and thanks for sharing the code. Would it be possible to get the pretrained models for the 3D shape manifold on ShapeNet?

Many thanks in advance, Luca

opened by lykius 7
Potential error in the code?

Hi, thanks for the great work and open source the code! However, when I look into the code, there seems to be some errors (or maybe I didn't understand it correctly). This part seems to duplicate with this part? I didn't see any reason for running the model twice here, so maybe it's an error?

I am also a bit confused about the code. I'm looking at the experiments where you train an IMNet for 3D shape (train_imnet_autodecoder.py). I follow the arguments you pass to the model and I think the model forward pass will enter this if-branch right? Under this if-branch, you are basically preparing things like linear combination weight out_dict['weights'] and distance between data and latent z latents_dist etc. for the LLE and Iso loss calculation. Is that correct?

One last thing is that, your model for 3D shape uses SIREN instead of fourier position encoding (the one from NeRF), while your model for image uses fourier instead of SIREN. Is it because the difference between 2D image space and 3D space? Did you find large difference in using these two modules e.g. fourier encoding didn't work for 3D shape experiment?

opened by Wuziyi616 3
Please fix the path in dataio.py

Hi Yilun,

Thanks for sharing the implementation of this awesome work! I found a path issue in dataio.py. Line: 578-581 if split == "train": self.im_paths = sorted(glob('/data/vision/billf/scratch/yilundu/dataset/celebahq/celebahq_train/data128x128_small/*.jpg')) else: self.im_paths = sorted(glob('/data/vision/billf/scratch/yilundu/dataset/celebahq/celebahq_test/*.jpg'))

Could you please help to fix it and share the dataset link so that we are based on the same dataset?

Thanks for your help.

opened by GH-HOME 1
Error in train_audio_autodecoder.py

Hey, while running the code to train audio manifolds, facing the following error in line 81 of train_audio_autodecoder.py-

train_generalization_dataset = dataio.AudioGeneralizationWrapper(train_audio_dataset, sampling=4096, do_pad=True) TypeError: __init__() got an unexpected keyword argument 'sampling'

The AudioGeneralizationWrapper given here, doesn't have the sampling argument.

opened by Aaryan369 3

[NeurIPS 2021] Code for Learning Signal-Agnostic Manifolds of Neural Fields

Related tags

Overview

Learning Signal-Agnostic Manifolds of Neural Fields

Downloading Data

Demo

Training Different Signal Manifolds

Citing our Paper

You might also like...

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

NeurIPS'21: Probabilistic Margins for Instance Reweighting in Adversarial Training (Pytorch implementation).

Code for CVPR 2021 paper: Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning

Code for our paper "Transfer Learning for Sequence Generation: from Single-source to Multi-source" in ACL 2021.

Code for EMNLP 2021 main conference paper "Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification"

Code for Findings at EMNLP 2021 paper: "Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning"

Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.

Easily train your own text-generating neural network of any size and complexity on any text dataset with a few lines of code.

code for "AttentiveNAS Improving Neural Architecture Search via Attentive Sampling"

Comments

pretrained models

Potential error in the code?

Please fix the path in dataio.py

Error in train_audio_autodecoder.py

Owner

Geometry-Consistent Neural Shape Representation with Implicit Displacement Fields

Data manipulation and transformation for audio signal processing, powered by PyTorch

Language-Agnostic SEntence Representations

MILES is a multilingual text simplifier inspired by LSBert - A BERT-based lexical simplification approach proposed in 2018. Unlike LSBert, MILES uses the bert-base-multilingual-uncased model, as well as simple language-agnostic approaches to complex word identification (CWI) and candidate ranking.

Official Pytorch implementation of Test-Agnostic Long-Tailed Recognition by Test-Time Aggregating Diverse Experts with Self-Supervision.

xFormers is a modular and field agnostic library to flexibly generate transformer architectures by interoperable and optimized building blocks.

Code to reprudece NeurIPS paper: Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks

Code for our paper "Mask-Align: Self-Supervised Neural Word Alignment" in ACL 2021

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

Code for the paper: Sequence-to-Sequence Learning with Latent Neural Grammars