Code for ICE-BeeM paper - NeurIPS 2020

Overview

ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA

This repository contains code to run and reproduce the experiments presented in ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA, published at NeurIPS 2020. Work done by Ilyes Khemakhem (Gatsby Unit, UCL), Ricardo Pio Monti (Gatsby Unit, UCL), Diederik P. Kingma (Google Research) and Aapo Hyvärinen (University of Helsinki).

Dependencies

This project was tested with the following versions:

  • python 3.6 and 3.7
  • pytorch 1.4
  • tensorflow 1.14
  • PyYAML 5.3.1
  • seaborn 0.10
  • scikit-learn 0.22.2
  • scipy 1.4.1

Running Simulations

We compared an ICE-BeeM model trained with flow contrastive estimation (FCE) to nonlinear ICA methods (iVAE and TCL).

We first compared these methods on nonstationary data generated according to a nonlinear ICA model (we refer to this dataset as TCL. Second, the data was generated from a nonstationary IMCA model (where the latent variables are dependent) which we refer to as IMCA.

To reproduce the simulations, run for e.g.:

python simulations.py --dataset IMCA --method icebeem

Type python simulations.py --help to learn about the arguments:

usage: simulations.py [-h] [--dataset DATASET] [--method METHOD]
                      [--config CONFIG] [--run RUN] [--nSims NSIMS] [--test]
                      [--plot]

optional arguments:
  -h, --help         show this help message and exit
  --dataset DATASET  Dataset to run experiments. Should be TCL or IMCA
  --method METHOD    Method to employ. Should be TCL, iVAE or ICE-BeeM
  --config CONFIG    Path to the config file
  --run RUN          Path for saving running related data.
  --nSims NSIMS      Number of simulations to run
  --test             Whether to evaluate the models from checkpoints
  --plot             Plot comparison of performances

The result of each simulation is saved in the value of the flag --run (defulat is run/ folder).

Running real data experiments

These experiments are run through the script main.py. Below are details on how to use the script. To learn about its arguments type python main.py --help:

usage: main.py [-h] [--config CONFIG] [--run RUN] [--n-sims N_SIMS]
               [--seed SEED] [--baseline] [--transfer] [--semisupervised]
               [--representation] [--mcc] [--second-seed SECOND_SEED]
               [--subset-size SUBSET_SIZE] [--all] [--plot]

optional arguments:
  -h, --help            show this help message and exit
  --config CONFIG       Path to the config file
  --run RUN             Path for saving running related data.
  --n-sims N_SIMS       Number of simulations to run
  --seed SEED           Random seed
  --baseline            Run the script for the baseline
  --transfer            Run the transfer learning experiments after
                        pretraining
  --semisupervised      Run semi-supervised experiments
  --representation      Run CCA representation validation across multiple
                        seeds
  --mcc                 compute MCCs -- only relevant for representation
                        experiments
  --second-seed SECOND_SEED
                        Second random seed for computing MCC -- only relevant
                        for representation experiments
  --subset-size SUBSET_SIZE
                        Number of data points per class to consider -- only
                        relevant for transfer learning if not run with --all
                        flag
  --all                 Run transfer learning experiment for many seeds and
                        subset sizes -- only relevant for transfer and
                        representation experiments
  --plot                Plot selected experiment for the selected dataset

All options and choice of dataset are passed through a configuration file under the configs folder. There are 4 main functions of interest:

  • train: trains a (conditional) energy model on the dataset specified by the configuration file, only considering the labels 0-n_labels, the latter being specified by the config file.
  • transfer: trains the secondary feature extractor g defining an ICE-BeeM on labels n_labels-len(dset) while keeping the feature extractor f fixed (after it was trained using train).
  • semisupervised: loads a feature extractor f from a (conditional) EBM pretrained on labels 0-n_labels and uses it to classify classes n_labels-len(dset)
  • compute_representations: trains a (conditional) energy model on a dataset (ignoring the n_labels field), saves the feature network f, and uses it to compute and save the learnt representation on the test split of the dataset.

Below, we explain how to call these functions using the main.py script to recreate the experiments from the manuscript. The experiments can be ran for:

  • MNIST: by setting the flag --config to mnist.yaml.
  • FashionMNIST: by setting the flag --config to fashionmnist.yaml.
  • CIFAR10: by setting the flag --config to cifar10.yaml.
  • CIFAR100: by setting the flag --config to cifar100.yaml.

The configuration files control most of the hyperparameters used for training and evaluation. Notable fields are:

  • final_layer: whether to apply a final FC layer to reduce the dimension of the latent space. To be used in conjunction with feature_size, which specifies the output (latent/feature) dimension.
  • architecture: the architecture of the feature extractor f, can be MLP, ConvMLP or Unet. These different architectures are discussed in the Appendix.
  • positive: whether to use positive features by applying a ReLU to the output of f (condition 3 of Theorem 2).
  • augment: whether to augment features by adding the square of f (condition 4 of Theorem 2). When changing these fields in configuration X.yaml, make sure to also change them in X_baseline.yaml. The latter only serves for the baseline of transfer learning.

Identifiability of representations

In these experiments we train multiple conditional and unconditional EBMs on various datasets and assess the identifiability of representations as discussed in Theorems 1-3.

These experiments therefore do the following:

  • Train conditional and unconditional EBMs using different random initializations on the full train split of the dataset.
  • Study the learnt representations over held out test data. We compare the MCC over held out representations as well as MCC after linear transformation using CCA (this is akin to weak identifiability).

To run the experiment on MNIST:


# train an ICE-BeeM on all labels, for seeds 0-9
# starting seed can be specified using --seed
python main.py --config mnist.yaml --representation --n-sims 10 
# train an unconditional EBM on all labels, for seeds 0-9
python main.py --config mnist.yaml --representation --n-sims 10 --baseline

Then, MCC statistics can be computed and visualized using:

# compute all pairwise MCCs for ICE-BeeM between seeds X and Y for X in {0..8} and Y in {X+1..9} 
# starting seed can be specified using --seed
python main.py --config mnist.yaml --representation --mcc --all --n-sims 10
# compute all pairwise MCCs for an unconditional EBM between seeds X and Y for X in {0..8} and Y in {X+1..9} 
python main.py --config mnist.yaml --representation --mcc --all --n-sims 10 --baseline

Finally, to visualize the MCC statistics with a boxplot:

# the number of seeds used for boxplot can be specified using --n-sims
# starting seed can be specified using --seed
python main.py --config mnist.yaml --representation --plot

Transfer learning

In this experiment, we compare:

  • Training an ICE-BeeM f.g on labels 0-7, then fixing f and learning only g on new unseen labels 8-9. This has the advantage of simplifying the learning, since g is much easier to train than f (we set it to a vector in our experiments).
  • Training an ICE-BeeM f.g directly on labels 8-9.

The idea is to see whether the feature extractor f can learn meaningful representations from similar datasets, especially when the size of the dataset is small. We use the denoising dcore matching (DSM) loss as a comparison metric (lower is better).

To run the experiment on MNIST:

# pretrain an ICE-BeeM on labels 0-7
python main.py --config mnist.yaml
# fix f and only learn g on labels 8-9 for many different dataset sizes and different seeds
python main.py --config mnist.yaml --transfer --all
# train an ICE-BeeM on labels 8-9 for many different dataset sizes and different seeds
python main.py --config mnist.yaml --transfer --baseline --all

The results are saved in the value of the flag --run (defulat is run/ folder). To plot the comparison for MNIST after running the steps above:

python main.py --config mnist.yaml --transfer --plot

Semi-supervised learning

In the semi-supervised learning experiment, we compare:

  • Training an ICE-BeeM f.g on labels 0-7, then using it to classify unseen labels 8-9.
  • Training an unconditional EBM h.1 on labels 0-7, then using it to classify unseen labels 8-9.

We use the classification accuracy as a comparison metric.

# pretrain an ICE-BeeM on labels 0-7 // same as first step in transfer learning exp
python main.py --config mnist.yaml
# pretrain an unconditional EBM on labels 0-7
python main.py --config mnist.yaml --baseline
# classify labels 8-9 using the pretrained ICE-BeeM -- outputs accuracy to stdout
python main.py --config mnist.yaml --semisupervised
# classify labels 8-9 using the pretrained unconditional EBM -- outputs accuracy to stdout
python main.py --config mnist.yaml --semisupervised --baseline

Using SLURM

The bash scripts slurm_main.sbatch (GPU) and slurm_main_cpu.sbatch (CPU) are SLURM wrapper around main.py and allow to run multiple experiments in parallel on a SLURM equipped server. You may have to change the #SBATCH configuration flags in the scripts according to your system.

These scripts set --n-sims to 1, and allows the user to select the seeds for which to run the experiments using the --array flag of sbatch. The rest of the arguments/flags can be passed as arguments of slurm_main.sbatch or slurm_main_cpu.sbatch:

sbatch --array=some_seed_values slurm_main.sbatch --the_rest_of_the_arguments

Examples

A use case is to run the transfer learning experiments in parallel:

sbatch --array=1-10 slurm_main.sbatch --config mnist.yaml --transfer --subset-size 500

This is equivalent to running:

python main.py --config mnist.yaml --seed x --transfer --subset-size 500

where x scans [1-10] for the value of the flag --seed. Following this approach requires to run the script for the flag --subset-size in [500, 1000, 2000, 3000, 4000, 5000, 6000].


Another use case for the identifiability of representations experiment is:

sbatch --array=42-51 slurm_main.sbatch --config mnist.yaml --representation

This is equivalent to:

python main.py --config mnist.yaml --seed 42 --n-sims 10 --representation

with the added advantage that all seeds are run in parallel.


The script can also be used to run single seeds: if the --array flag is not set, then the seed value can be passed as an argument --seed to slurm_main.batch:

sbatch slurm_main.sbatch --config mnist.yaml --seed 42

is equivalent to

python main.py --config mnist.yaml --seed 42

Run everything using SLURM

To maximize the use of parallel computing, we can get around the --all flag which launches loops inside the main.py script by looping manually around the SLURM script as shown below. To run everything, you can use the following two step bash script:

CONFIG_FILE=mnist.yaml
sbatch slurm_main.sbatch --config $CONFIG_FILE
sbatch slurm_main.sbatch --config $CONFIG_FILE  --baseline
for SIZE in 0 500 1000 2000 3000 4000 5000 6000; do
        sbatch --array=0-4 slurm_main.sbatch --config $CONFIG_FILE --transfer --subset-size $SIZE --baseline
done
sbatch --array=0-19 slurm_main.sbatch --config $CONFIG_FILE  --representation
sbatch --array=0-19 slurm_main.sbatch --config $CONFIG_FILE  --representation --baseline

After these jobs finish and save the necessary weights and tensors, run:

CONFIG_FILE=mnist.yaml
typeset -i SEED SECSEED
for SEED in {0..18}; do
    for ((SECSEED=SEED+1;SECSEED<=19;SECSEED++)); do
        sbatch slurm_main_cpu.sbatch --config $CONFIG_FILE --representation --mcc --seed $SEED --second-seed $SECSEED
        sbatch slurm_main_cpu.sbatch --config $CONFIG_FILE --representation --mcc --seed $SEED --second-seed $SECSEED --baseline
    done
done
for SIZE in 0 500 1000 2000 3000 4000 5000 6000; do
        sbatch --array=0-4 slurm_main.sbatch --config $CONFIG_FILE --transfer --subset-size $SIZE
done
sbatch slurm_main.sbatch --config $CONFIG_FILE  --semisupervised
sbatch slurm_main.sbatch --config $CONFIG_FILE  --semisupervised --baseline

References

If you find this code helpful/inspiring for your research, we would be grateful if you cite the following:

@inproceedings{khemakhem2020ice,
 author = {Khemakhem, Ilyes and Monti, Ricardo and Kingma, Diederik and Hyvarinen, Aapo},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {12768--12778},
 publisher = {Curran Associates, Inc.},
 title = {ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA},
 url = {https://proceedings.neurips.cc/paper/2020/file/962e56a8a0b0420d87272a682bfd1e53-Paper.pdf},
 volume = {33},
 year = {2020}
}

License

A full copy of the license can be found here.

Copyright (C) 2020 Ilyes Khemakhem

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <https://www.gnu.org/licenses/>.
Comments
  • Theory vs implementation

    Theory vs implementation

    @ilkhem @piomonti

    1. In fce.py, the value for the noise segment label is set as 1/number_of_segments: self.contrastSegments = (np.ones(self.segments.shape) / self.segments.shape[1]).astype(np.float32) I could not find an explanation for this choice in the paper. Could you clarify? Also, would you have a different recommendation if the number_of_observations were different for each segment?

    2. In compute_ebm_logpdf, the augmented evaluation of f(x).T * g(y) does not seem to work for noise samples. My understanding is that here both f(x) and g(y) are augmented with f^2 and g^2. If that is the case, then g^2 = (W*y)^2, where * means matrix multiplication, squaring is element-wise and W = self.ebm_finalLayer. When y = one-hot encoded labels of data, g^2 = W[:,k]^2 where one-hot selects the k-th col of W which is then squared. This is equivalent to the implementation in line 63: self.ebm_finalLayer * self.ebm_finalLayer (this is just element-wise squaring of W) followed by * seg in line 66 for column selection (roughly speaking). The problem is with noise samples, for which y = self.contrastSegments instead of one-hot encoded labels. Then g^2 = mean(W,1)^2, the mean of W cols... and the MEANS (not the elements of W) are squared. So the current implementation in line 63: self.ebm_finalLayer * self.ebm_finalLayer does not seem to work for noise samples. Could you provide a clarification on this?

    Thank you for your time and for the outstanding work!

    Rogers

    opened by rsilva8 0
  • Moredataset?

    Moredataset?

    Thanks for the excellent work. I just wanted to know if you have worked on some popular datasets as well, such as 3dshapes, smallnorb and cars3d. If so, that could be helpful to merge the benchmark with disentanglement_lib.

    Besides, as the conditional case requires the factor label to train the model, how could that still be "unsupervised"?

    opened by noahcao 0
  • Could you explain a little on what does parameter annealing do?

    Could you explain a little on what does parameter annealing do?

    https://github.com/ilkhem/icebeem/blob/0077f0120c83bcc6d9b199b831485c42bed2401f/models/ivae/ivae_core.py#L255

    It seems this part was not mentioned in the AISTATS'20 paper, is this trick important in terms of performance? could you briefly clarify what this part does?

    opened by rguo12 0
  • How do I get plot of true v/s predicted latent dimension

    How do I get plot of true v/s predicted latent dimension

    Hi,

    I use the simulations.py for running the simulated experiment using imca dataset but I am not able to get the plot of the true v/s latent dimension using the arguments to be passed. Could you help shed light on the way to get the plot of true v/s latent dimensions.

    --Prashant

    opened by prashkmr 0
  • fMRI component analysis

    fMRI component analysis

    Great repo! My name is Yerzhan, I am a masters student at Skoltech, Moscow. I am doing research at a neuroimaging lab here, and we are looking for the ways to perform rs-fMRI component analysis using deep learning methods.

    Can you please comment on application of iVAE and TCL to fMRI component analysis. It is briefly mentioned in the paper, yet I was wondering if you can elaborate more on results that you obtained. How good they were, and perhaps there are some reproducible code of model being trained on HCP data. That would be of tremendous help.

    Thank you!

    opened by yerzhanimanmalik 0
Code for Discriminative Sounding Objects Localization (NeurIPS 2020)

Discriminative Sounding Objects Localization Code for our NeurIPS 2020 paper Discriminative Sounding Objects Localization via Self-supervised Audiovis

null 51 Dec 11, 2022
Code for the Population-Based Bandits Algorithm, presented at NeurIPS 2020.

Population-Based Bandits (PB2) Code for the Population-Based Bandits (PB2) Algorithm, from the paper Provably Efficient Online Hyperparameter Optimiza

Jack Parker-Holder 22 Nov 16, 2022
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

UDP-Pose This is the pytorch implementation for UDP++, which won the Fisrt place in COCO Keypoint Challenge at ECCV 2020 Workshop. Top-Down Results on

null 20 Jul 29, 2022
Code for paper ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization in the Loop.

Who Left the Dogs Out? Evaluation and demo code for our ECCV 2020 paper: Who Left the Dogs Out? 3D Animal Reconstruction with Expectation Maximization

Benjamin Biggs 29 Dec 28, 2022
git《Beta R-CNN: Looking into Pedestrian Detection from Another Perspective》(NeurIPS 2020) GitHub:[fig3]

Beta R-CNN: Looking into Pedestrian Detection from Another Perspective This is the pytorch implementation of our paper "[Beta R-CNN: Looking into Pede

null 35 Sep 8, 2021
Official implementation of "GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators" (NeurIPS 2020)

GS-WGAN This repository contains the implementation for GS-WGAN: A Gradient-Sanitized Approach for Learning Differentially Private Generators (NeurIPS

null 46 Nov 9, 2022
Diverse Image Captioning with Context-Object Split Latent Spaces (NeurIPS 2020)

Diverse Image Captioning with Context-Object Split Latent Spaces This repository is the PyTorch implementation of the paper: Diverse Image Captioning

Visual Inference Lab @TU Darmstadt 34 Nov 21, 2022
Official implementation for Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder at NeurIPS 2020

Likelihood-Regret Official implementation of Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder at NeurIPS 2020. T

Xavier 33 Oct 12, 2022
Official Pytorch implementation of 'GOCor: Bringing Globally Optimized Correspondence Volumes into Your Neural Network' (NeurIPS 2020)

Official implementation of GOCor This is the official implementation of our paper : GOCor: Bringing Globally Optimized Correspondence Volumes into You

Prune Truong 71 Nov 18, 2022
《Dual-Resolution Correspondence Network》(NeurIPS 2020)

Dual-Resolution Correspondence Network Dual-Resolution Correspondence Network, NeurIPS 2020 Dependency All dependencies are included in asset/dualrcne

Active Vision Laboratory 45 Nov 21, 2022
(NeurIPS 2020) Wasserstein Distances for Stereo Disparity Estimation

Wasserstein Distances for Stereo Disparity Estimation Accepted in NeurIPS 2020 as Spotlight. [Project Page] Wasserstein Distances for Stereo Disparity

Divyansh Garg 92 Dec 12, 2022
Official Implementation of Swapping Autoencoder for Deep Image Manipulation (NeurIPS 2020)

Swapping Autoencoder for Deep Image Manipulation Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros, Richard Zhang UC

null 449 Dec 27, 2022
Implementation of "Fast and Flexible Temporal Point Processes with Triangular Maps" (Oral @ NeurIPS 2020)

Fast and Flexible Temporal Point Processes with Triangular Maps This repository includes a reference implementation of the algorithms described in "Fa

Oleksandr Shchur 20 Dec 2, 2022
[NeurIPS 2020] Blind Video Temporal Consistency via Deep Video Prior

pytorch-deep-video-prior (DVP) Official PyTorch implementation for NeurIPS 2020 paper: Blind Video Temporal Consistency via Deep Video Prior TensorFlo

Yazhou XING 90 Oct 19, 2022
Advances in Neural Information Processing Systems (NeurIPS), 2020.

What is being transferred in transfer learning? This repo contains the code for the following paper: Behnam Neyshabur*, Hanie Sedghi*, Chiyuan Zhang*.

Google Research 36 Aug 26, 2022
Neuron Merging: Compensating for Pruned Neurons (NeurIPS 2020)

Neuron Merging: Compensating for Pruned Neurons Pytorch implementation of Neuron Merging: Compensating for Pruned Neurons, accepted at 34th Conference

Woojeong Kim 33 Dec 30, 2022
Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

Xin Liu 106 Dec 30, 2022
Defending graph neural networks against adversarial attacks (NeurIPS 2020)

GNNGuard: Defending Graph Neural Networks against Adversarial Attacks Authors: Xiang Zhang ([email protected]), Marinka Zitnik (marinka@hms.

Zitnik Lab @ Harvard 44 Dec 7, 2022
[NeurIPS 2020] Official repository for the project "Listening to Sound of Silence for Speech Denoising"

Listening to Sounds of Silence for Speech Denoising Introduction This is the repository of the "Listening to Sounds of Silence for Speech Denoising" p

Henry Xu 40 Dec 20, 2022