Code for MarioNette: Self-Supervised Sprite Learning, in NeurIPS 2021

Dima Smirnov

Last update: Nov 18, 2022

Related tags

Deep Learning MarioNette

Overview

MarioNette | Webpage | Paper | Video

MarioNette: Self-Supervised Sprite Learning
Dmitriy Smirnov, Michaël Gharbi, Matthew Fisher, Vitor Guizilini, Alexei A. Efros, Justin Solomon
NeurIPS 2021

Set-up

To install the neecssary dependencies, run:

conda env create -f environment.yml
conda activate MarioNette

Also, be sure to execute export PYTHONPATH=:$PYTHONPATH prior to running any of the scripts.

Training

To train a MarioNette model, run:

python scripts/train.py --checkpoint_dir out_dir --data data_dir

Your dataset should be stored in data_dir, with each input frame named #.png. If the images are not 128x128 pixels, specify the resolution using the --canvas_size flag. Optionally, pass a --layer_size flag to specify the anchor grid resolution, --num_layers to specify the number of layers, or --num_classes to specify the size of the spirte dictionary.

To monitor the training, launch a TensorBoard instance with --logdir out_dir.

BibTeX

@article{smirnov2021marionette,
  title={{MarioNette}: Self-Supervised Sprite Learning},
  author={Smirnov, Dmitriy and Gharbi, Michael and Fisher, Matthew and Guizilini, Vitor and Efros, Alexei A. and Solomon, Justin},
  year={2021},
  journal={Conference on Neural Information Processing Systems}
}

Comments

Guidance with SpaceInvaders

Hi folks,

I trained MarioNette with a data set of 2^15 128x128 scenes of SpaceInvaders, but only the background is reconstructed. Do you have some ideas, about which kind of parameters I should modify to reach the results described in the paper?

Thanks a lot for your insights!

opened by ThomasRot 4
Game datasets used in the paper?

Hi, Dmitriy,

Congrats on the nice work !!! Could you share the game datasets used in the paper? (Fighting Hero (one level, 5,330 frames); Nintendo Super Mario Bros. (one level, 2,220 frames) and ATARI Space Invaders (5,000 frames))

Best,

Xi

opened by XiSHEN0220 2

Implementation of Self-supervised Graph-level Representation Learning with Local and Global Structure (ICML 2021).

Self-supervised Graph-level Representation Learning with Local and Global Structure Introduction This project is an implementation of ``Self-supervise

50 Dec 9, 2022

Self-Supervised Pillar Motion Learning for Autonomous Driving (CVPR 2021)

Self-Supervised Pillar Motion Learning for Autonomous Driving Chenxu Luo, Xiaodong Yang, Alan Yuille Self-Supervised Pillar Motion Learning for Autono

101 Dec 5, 2022

[EMNLP 2021] Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

RoSTER The source code used for Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training, p

60 Dec 30, 2022

Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection, AAAI 2021.

Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection This repository is an official implementation of the AAAI 2021 paper Co-mi

20 Dec 7, 2022

[BMVC 2021] Official PyTorch Implementation of Self-supervised learning of Image Scale and Orientation Estimation

Self-Supervised Learning of Image Scale and Orientation Estimation (BMVC 2021) This is the official implementation of the paper "Self-Supervised Learn

17 Nov 10, 2022

Official implementation of NeurIPS 2021 paper "Contextual Similarity Aggregation with Self-attention for Visual Re-ranking"

CSA: Contextual Similarity Aggregation with Self-attention for Visual Re-ranking PyTorch training code for CSA (Contextual Similarity Aggregation). We

19 Oct 21, 2022

$Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\$

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)

Skyformer This repository is the official implementation of Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Method (NeurIPS 2021).

46 Sep 20, 2022

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021)

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021) The implementation of Reducing Infromation Bottleneck for W

81 Dec 16, 2022

[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.

TBE The source code for our paper "Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Le

150 Dec 28, 2022

Code for MarioNette: Self-Supervised Sprite Learning, in NeurIPS 2021

Related tags

Overview

MarioNette | Webpage | Paper | Video

Set-up

Training

BibTeX

You might also like...

Implementation of Self-supervised Graph-level Representation Learning with Local and Global Structure (ICML 2021).

Self-Supervised Pillar Motion Learning for Autonomous Driving (CVPR 2021)

[EMNLP 2021] Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training

Co-mining: Self-Supervised Learning for Sparsely Annotated Object Detection, AAAI 2021.

[BMVC 2021] Official PyTorch Implementation of Self-supervised learning of Image Scale and Orientation Estimation

Official implementation of NeurIPS 2021 paper "Contextual Similarity Aggregation with Self-attention for Visual Re-ranking"

Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr\"om Method (NeurIPS 2021)

Reducing Information Bottleneck for Weakly Supervised Semantic Segmentation (NeurIPS 2021)

[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.

Comments

Guidance with SpaceInvaders

Game datasets used in the paper?

Owner

Dima Smirnov

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

[CVPR 2021] "The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models" Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, Zhangyang Wang

Code for the paper "Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds" (ICCV 2021)

Dense Contrastive Learning (DenseCL) for self-supervised representation learning, CVPR 2021.

Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

Code for our ACL 2021 paper - ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.