[NeurIPS 2021] Code for Unsupervised Learning of Compositional Energy Concepts

Related tags

Deep Learning comet
Overview

Unsupervised Learning of Compositional Energy Concepts

This is the pytorch code for the paper Unsupervised Learning of Compositional Energy Concepts.

Demo

Please download a pretrained model at this link and then execute the following code to test a pretrained CelebA-HQ 128x128 COMET model

python demo.py im_path=im0.png

Global Factor Decomposition

Please utilize the following command to run global factor decomposition on CelebA-HQ (or other datasets)

python train.py --exp=celebahq --batch_size=12 --gpus=1 --cuda --train --dataset=celebahq --step_lr=500.0

You may further run the code on high-resolution 128x128 images below

python train.py --exp=celebahq_128 --batch_size=12 --gpus=1 --cuda --train --dataset=celebahq_128 --step_lr=500.0

Local Factor Decomposition

Please utilize the following command to run local factor decomposition on CLEVR

python train.py --exp=clevr_local_decomp --num_steps=5 --step_lr=1000.0 --components=4 --dataset=clevr --cuda --train --batch_size=24 --latent_dim=16 --recurrent_model --pos_embed

Dataset Download

Please utilize the following link to download the CLEVR dataset utilized in our experiments. Downloads for additional datasets will be posted soon. Feel free to raise an issue if there is a particular dataset you would like downloaded

Citing our Paper

If you find our code useful for your research, please consider citing

@inproceedings{du2021comet,
  title={Unsupervised Learning of Compositional Energy Concepts},
  author={Du, Yilun and Li, Shuang and Sharma, Yash and Tenenbaum, B. Joshua
  and Mordatch, Igor},
  booktitle={Advances in Neural Information Processing Systems},
  year={2021}
}
You might also like...
Compositional and Parameter-Efficient Representations for Large Knowledge Graphs

NodePiece - Compositional and Parameter-Efficient Representations for Large Knowledge Graphs NodePiece is a "tokenizer" for reducing entity vocabulary

Compositional Sketch Search
Compositional Sketch Search

Compositional Sketch Search Official repository for ICIP 2021 Paper: Compositional Sketch Search Requirements Install and activate conda environment c

A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

CLEVR Dataset Generation This is the code used to generate the CLEVR dataset as described in the paper: CLEVR: A Diagnostic Dataset for Compositional

ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

The ImageNet-CoG Benchmark Project Website Paper (arXiv) Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalizatio

Face Mask Detection System built with OpenCV, TensorFlow using Computer Vision concepts

Face mask detection Face Mask Detection System built with OpenCV, TensorFlow using Computer Vision concepts in order to detect face masks in static im

Image Segmentation Animation using Quadtree concepts.
Image Segmentation Animation using Quadtree concepts.

QuadTree Image Segmentation Animation using QuadTree concepts. Usage usage: quad.py [-h] [-fps FPS] [-i ITERATIONS] [-ws WRITESTART] [-b] [-img] [-s S

Python Classes: Medical Insurance Project using Object Oriented Programming Concepts

Medical-Insurance-Project-OOP Python Classes: Medical Insurance Project using Object Oriented Programming Concepts Classes are an incredibly useful pr

the code for paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration"

EOW-Softmax This code is for the paper "Energy-Based Open-World Uncertainty Modeling for Confidence Calibration". Accepted by ICCV21. Usage Commnd exa

PyTorch code accompanying our paper on Maximum Entropy Generators for Energy-Based Models

Maximum Entropy Generators for Energy-Based Models All experiments have tensorboard visualizations for samples / density / train curves etc. To run th

Comments
  • Question regarding to training CelebAHQ_128x128

    Question regarding to training CelebAHQ_128x128

    Hello Thanks for your great work. I have question about line 360-387 in train.py. If we run the command in the doc directly, line 387 causes error because vgg_loss is not used when the iteration is less than 10000.

                if it < 10000 or FLAGS.dataset != "celebahq_128":
                    loss = im_loss
                else:
                    vgg_loss = loss_fn_vgg(im_negs[:, -1], im).mean()
                    loss = vgg_loss  + 0.1 * im_loss
    
                loss.backward()
                if FLAGS.gpus > 1:
                    average_gradients(models)
    
                [torch.nn.utils.clip_grad_norm_(model.parameters(), 10.0) for model in models]
                [optimizer.step() for optimizer in optimizers]
                [optimizer.zero_grad() for optimizer in optimizers]
    
                if it % FLAGS.log_interval == 0 and rank_idx == 0:
                    loss = loss.item()
                    energy_pos_mean = energy_pos.mean().item()
                    energy_neg_mean = energy_neg.mean().item()
                    energy_pos_std = energy_pos.std().item()
                    energy_neg_std = energy_neg.std().item()
    
                    kvs = {}
                    kvs['loss'] = loss
                    kvs['ml_loss'] = ml_loss.item()
                    kvs['im_loss'] = im_loss.item()
    
                    if FLAGS.dataset == "celebahq_128":
                        kvs['vgg_loss'] = vgg_loss.item()
    

    Another question is that why do we add vgg_loss to im_loss when the iteration pass 10000? Is there any intuition or purpose behind this design? (Can we use vgg loss directly at the very beginning?) Thanks

    opened by ChenChengKuan 2
  • Experiments setup for object-level decomposition on Tetris

    Experiments setup for object-level decomposition on Tetris

    Hi, thank you for presenting such an interesting paper and neat code. I am trying to reproduce the object-level decomposition on Tetris dataset. The decomposition result is not ideal when I use the same settings as the CLEVR dataset: python train.py --exp=tetris_local_decomp --num_steps=5 --step_lr=1000.0 --components=3 --dataset=tetris --cuda --train --batch_size=24 --latent_dim=16 --recurrent_model --pos_embed Would you mind providing more details of the experiment setup for object-level decomposition on Tetris? Thank you very much.

    opened by YangYafei1998 1
  • About GT Masks

    About GT Masks

    Hi Yilun, I am very interesting in your work. I find that the clver dataset you provide is different from https://github.com/deepmind/multi_object_datasets#clevr-with-masks or https://cs.stanford.edu/people/jcjohns/clevr/. Therefore, I can not download the groundtruth masks from these two links. And the groundtruth masks are also not provided in your link. Can you also provide me the the groundtruth masks of your dataset? Thanks, Tao

    opened by ThomasMrY 4
Owner
null
This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

null 1.1k Dec 30, 2022
The first machine learning framework that encourages learning ML concepts instead of memorizing class functions.

SeaLion is designed to teach today's aspiring ml-engineers the popular machine learning concepts of today in a way that gives both intuition and ways of application. We do this through concise algorithms that do the job in the least jargon possible and examples to guide you through every step of the way.

Anish 324 Dec 27, 2022
[CVPR 2021] A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Visual-Reasoning-eXplanation [CVPR 2021 A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts] Project Page | Vid

Andy_Ge 54 Dec 21, 2022
Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020).

SentiBERT Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020). https://arxiv.org/abs/20

Da Yin 66 Aug 13, 2022
Code to reproduce the results for Compositional Attention: Disentangling Search and Retrieval.

Compositional-Attention This repository contains the official implementation for the paper Compositional Attention: Disentangling Search and Retrieval

Sarthak Mittal 17 Oct 23, 2021
[CVPR'22] COAP: Learning Compositional Occupancy of People

COAP: Compositional Articulated Occupancy of People Paper | Video | Project Page This is the official implementation of the CVPR 2022 paper COAP: Lear

Marko Mihajlovic 111 Dec 11, 2022
ManimML is a project focused on providing animations and visualizations of common machine learning concepts with the Manim Community Library.

ManimML ManimML is a project focused on providing animations and visualizations of common machine learning concepts with the Manim Community Library.

null 259 Jan 4, 2023
Dense Unsupervised Learning for Video Segmentation (NeurIPS*2021)

Dense Unsupervised Learning for Video Segmentation This repository contains the official implementation of our paper: Dense Unsupervised Learning for

Visual Inference Lab @TU Darmstadt 173 Dec 26, 2022
[NeurIPS 2021] ORL: Unsupervised Object-Level Representation Learning from Scene Images

Unsupervised Object-Level Representation Learning from Scene Images This repository contains the official PyTorch implementation of the ORL algorithm

Jiahao Xie 55 Dec 3, 2022
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020