Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun

Junbo (Jake) Zhao

Last update: Jan 2, 2023

Related tags

Deep Learning ARAE

Overview

ARAE

Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun https://arxiv.org/abs/1706.04223

Disclaimer

Major updates on 06/11/2018:

WGAN-GP replaced WGAN
added 1BWord dataset experiment
added Yelp transfer experiment
removed unnecessary tricks
added both RNNLM and ngram-LM evaluation for both forward and reverse PPL.

File structure

lang: ARAE for language generation, on both 1B word benchmark and SNLI
yelp: ARAE for language style transfer
mnist (in Torch): ARAE for discretized MNIST

Reference

@ARTICLE{2017arXiv170604223J,
   author = {{Junbo} and {Zhao} and {Kim}, Y. and {Zhang}, K. and {Rush}, A.~M. and 
	{LeCun}, Y.},
    title = "{Adversarially Regularized Autoencoders for Generating Discrete Structures}",
  journal = {ArXiv e-prints},
archivePrefix = "arXiv",
   eprint = {1706.04223},
 primaryClass = "cs.LG",
 keywords = {Computer Science - Learning, Computer Science - Computation and Language, Computer Science - Neural and Evolutionary Computing},
     year = 2017,
    month = jun,
   adsurl = {http://adsabs.harvard.edu/abs/2017arXiv170604223J},
  adsnote = {Provided by the SAO/NASA Astrophysics Data System}
}

Comments

Runtime error
conda create -n pytorch python=3.5 anaconda source activate pytorch conda install pytorch torchvision cuda80 -c soumith python train.py --data_path PATH_TO_PROCESSED_DATA --cuda --kenlm_path PATH_TO_KENLM_DIRECTORY

Training... Traceback (most recent call last): File "train.py", line 516, in train_ae(train_data[niter], total_loss_ae, start_time, niter) File "train.py", line 352, in train_ae output = autoencoder(source, lengths, noise=True) File "/home/$USER/anaconda2/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in call result = self.forward(*input, **kwargs) File "/home/$USER/workspace/ARAE/pytorch/models.py", line 174, in forward hidden = self.encode(indices, lengths, noise) File "/home/$USER/workspace/ARAE/pytorch/models.py", line 202, in encode hidden = torch.div(hidden, norms.expand_as(hidden)) File "/home/$USER/anaconda2/envs/pytorch/lib/python3.5/site-packages/torch/autograd/variable.py", line 725, in expand_as return Expand.apply(self, (tensor.size(),)) File "/home/$USER/anaconda2/envs/pytorch/lib/python3.5/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward result = i.expand(*new_size) RuntimeError: The expanded size of the tensor (300) must match the existing size (64) at non-singleton dimension 1. at /opt/conda/conda-bld/pytorch_1502008109146/work/torch/lib/THC/generic/THCTensor.c:323
opened by shaform 4
Loading model with low accuracy

Hi! I found that when I load the model back using autoencoder.load_state_dict(torch.load(ae_path)) etc., it leads to low accuracy (around 0.3) even though the model achieves around 0.8 accuracy in evaluate_autoencoder during training. I have to use torch.load(ae_path) and torch.save(autoencoder, f)to get away this problem. Probably it is a pytorch bug, as discussed here: https://discuss.pytorch.org/t/saving-and-loading-a-model-in-pytorch/2610/6

opened by posenhuang 3

IndexError for last batch of last epoch

I got this error while training a full model yesterday:

Traceback (most recent call last):
  File "train.py", line 526, in <module>
    train_gan_d(train_data[random.randint(0, niter)])
IndexError: list index out of range

This happened for the last batch of the last epoch.

The relevant code block:

# train gan ----------------------------------
        for k in range(niter_gan):

            # train discriminator/critic
            for i in range(args.niters_gan_d):
                # feed a seen sample within this epoch; good for early training
                errD, errD_real, errD_fake = \
                    train_gan_d(train_data[random.randint(0, niter)])

It is possible that train_data can be populated with less elements than niter in the batchifyfunction?

The argument list I had was pretty minimal: python train.py --data_path /path/to/billion-word-data --cuda --no_earlystopping

opened by thvasilo 2

errG_real is not backpropagated in the training of generator

According to the Algorithm1 in the paper, L_G=errG_real-errG_fake. However, in your pytorch implementation, only errG_fake is backpropagated. Why? How does it affect the performance if you backpropagate both?

opened by guxd 0
[snli] pretrained weight works while training from scratch results in very bad result.

I found that the provided weights work quite well when I generate samples. However, if I train the model with the provided script(the same setting), the result is very pool. Are there any changes in the code?

opened by yanghoonkim 0
Are there something wrong with the gan_d_loss?

I found that in the function train_gan_d(), the loss_fake and loss_real are in the opposite direction with the original paper. I was wondering is it correct or not?

opened by shizhediao 0
How to use "offset vector transformations" to change text like in the paper?

I am wondering how to use this code to perform the "offset vector transformation" examples in the paper, such as substituting "clapping" with "walking" and altering the entire input sentence to fit the new word.

opened by styfeng 0

Code for the paper "Adversarially Regularized Autoencoders (ICML 2018)" by Zhao, Kim, Zhang, Rush and LeCun

Related tags

Overview

ARAE

Disclaimer

File structure

Reference

Comments

Runtime error

Loading model with low accuracy

IndexError for last batch of last epoch

errG_real is not backpropagated in the training of generator

[snli] pretrained weight works while training from scratch results in very bad result.

Are there something wrong with the gan_d_loss?

How to use "offset vector transformations" to change text like in the paper?

Owner

Junbo (Jake) Zhao

Code for the ICML 2021 paper "Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation", Haoxiang Wang, Han Zhao, Bo Li.

The implementation of CVPR2021 paper Temporal Query Networks for Fine-grained Video Understanding, by Chuhan Zhang, Ankush Gupta and Andrew Zisserman.

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

[CVPRW 21] "BNN - BN = ? Training Binary Neural Networks without Batch Normalization", Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, Zhangyang Wang

Y. Zhang, Q. Yao, W. Dai, L. Chen. AutoSF: Searching Scoring Functions for Knowledge Graph Embedding. IEEE International Conference on Data Engineering (ICDE). 2020

[Preprint] "Chasing Sparsity in Vision Transformers: An End-to-End Exploration" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang

Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Graph Convolutional Networks for Hyperspectral Image Classification, IEEE TGRS, 2021.

[CVPR 2022] "The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy" by Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, Zhangyang Wang

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Source code, datasets and trained models for the paper Learning Advanced Mathematical Computations from Examples (ICLR 2021), by François Charton, Amaury Hayat (ENPC-Rutgers) and Guillaume Lample

Code and data of the Fine-Grained R2R Dataset proposed in paper Sub-Instruction Aware Vision-and-Language Navigation

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)