Learning to Simulate Dynamic Environments with GameGAN (CVPR 2020)

Related tags

Deep Learning GameGAN_code
Overview

Learning to Simulate Dynamic Environments with GameGAN

PyTorch code for GameGAN

Learning to Simulate Dynamic Environments with GameGAN
Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, Sanja Fidler
CVPR, 2020
[Paper] [Project Page] [Video]

Abstract: Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN "renders" the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist.

Citation

  • If you found this codebase useful in your research, please cite:
@inproceedings{kim2020learning,
  title={Learning to Simulate Dynamic Environments with GameGAN},
  author={Kim, Seung Wook and Zhou, Yuhao and Philion, Jonah and Torralba, Antonio and Fidler, Sanja},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={1231--1240},
  year={2020}
}

Environment Setup

This codebase is tested with Ubuntu 18.04 and python3.8.3

  • Clone the repository
git clone https://github.com/nv-tlabs/GameGAN_code.git
cd GameGAN_code
  • Install dependencies
pip install -r requirements.txt

Dataset extraction

Please clone and follow https://github.com/hardmaru/WorldModelsExperiments/tree/master/doomrnn to install the VizDoom environment.

  • Copy extraction scripts and run
cp data/extract.py DOOMRNN_DIR/
cp data/extract_data.sh DOOMRNN_DIR/
cd DOOMRNN_DIR
./extract_data.sh
  • Now, extracted data is saved in 'DOOMRNN_DIR/vizdoom_skip3'
cd GameGAN_code/data
python dataloader.py DOOMRNN_DIR/vizdoom_skip3 vizdoom
  • You should now see .npy files extracted in 'data/vizdoom' directory.

For custom datasets, you can construct .npy files that contain a sequence of image and action pairs and define a dataloader similar to 'class vizdoom_dataset'. Please refer to 'data/dataloder.py'.

-- The above repository is deprecated and VizDoom environment might not run correctly in certain systems. In that case, you can use the docker installtaion of https://github.com/zacwellmer/WorldModels and copy the extraction scripts in the docker environment.

Training

We provide training scripts in './scripts'.

  • For training the full GameGAN model, run:
./scripts/vizdoom_multi.sh
  • For training the GameGAN model without the external memory module, run:
./scripts/vizdoom_single.sh

Monitoring

  • You can monitor the training process with tensorboard:
tensorboard --port=PORT --logdir=./results

Tips

  • Different environments might need different hyper-parameters. The most important hyper-parameter is 'recon_loss_multiplier' in 'config.py', which usually works well with 0.001 ~ 0.05.
  • Environments that do not need long-term consistency usually works better without the external memory module
You might also like...
PyBullet CartPole and Quadrotor environments—with CasADi symbolic a priori dynamics—for learning-based control and reinforcement learning
PyBullet CartPole and Quadrotor environments—with CasADi symbolic a priori dynamics—for learning-based control and reinforcement learning

safe-control-gym Physics-based CartPole and Quadrotor Gym environments (using PyBullet) with symbolic a priori dynamics (using CasADi) for learning-ba

Dynamic Systems Lab 300 Dec 28, 2022
An official implementation of
An official implementation of "SFNet: Learning Object-aware Semantic Correspondence" (CVPR 2019, TPAMI 2020) in PyTorch.

PyTorch implementation of SFNet This is the implementation of the paper "SFNet: Learning Object-aware Semantic Correspondence". For more information,

CV Lab @ Yonsei University 87 Dec 30, 2022
Official PyTorch code for CVPR 2020 paper
Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision https://arxiv.org/abs/2003.00393 Abstract Active learning (AL) aims to min

Denis 29 Nov 21, 2022
Implementation of CVPR 2020 Dual Super-Resolution Learning for Semantic Segmentation
Implementation of CVPR 2020 Dual Super-Resolution Learning for Semantic Segmentation

Dual super-resolution learning for semantic segmentation 2021-01-02 Subpixel Update Happy new year! The 2020-12-29 update of SISR with subpixel conv p

Sam 79 Nov 24, 2022
This repository contains the code for the CVPR 2020 paper
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 697 Jan 6, 2023
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).
UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

UDP-Pose This is the pytorch implementation for UDP++, which won the Fisrt place in COCO Keypoint Challenge at ECCV 2020 Workshop. Top-Down Results on

null 20 Jul 29, 2022
PyTorch implementation of paper
PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

Neural Scene Flow Fields PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 20

Zhengqi Li 585 Jan 4, 2023
(CVPR 2021) PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds
(CVPR 2021) PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds by Mutian Xu*, Runyu Ding*, Hengshuang Zhao, and Xiaojuan Qi. Int

CVMI Lab 228 Dec 25, 2022
Dynamic Slimmable Network (CVPR 2021, Oral)
Dynamic Slimmable Network (CVPR 2021, Oral)

Dynamic Slimmable Network (DS-Net) This repository contains PyTorch code of our paper: Dynamic Slimmable Network (CVPR 2021 Oral). Architecture of DS-

Changlin Li 197 Dec 9, 2022
Comments
  • Deprecated dependencies

    Deprecated dependencies

    This repository is new and comes from impressive and innovative research beyond what my words can express. I'm very interested in testing this to better understand the subject and maybe have a participation in it in the future, but the instructions here to obtain training data refer to another older repository which depends on doompy, a badly deprecated package that fails to build in current python environments with this error log:

    2020-12-28T13:56:50,722 Created temporary directory: /tmp/pip-ephem-wheel-cache-1bnxczyv 2020-12-28T13:56:50,723 Created temporary directory: /tmp/pip-req-tracker-1o5ngo8p 2020-12-28T13:56:50,723 Created requirements tracker '/tmp/pip-req-tracker-1o5ngo8p' 2020-12-28T13:56:50,724 Created temporary directory: /tmp/pip-install-cy882_jk 2020-12-28T13:56:50,800 1 location(s) to search for versions of doom-py: 2020-12-28T13:56:50,800 * https://pypi.org/simple/doom-py/ 2020-12-28T13:56:50,800 Getting page https://pypi.org/simple/doom-py/ 2020-12-28T13:56:50,801 Found index url https://pypi.org/simple 2020-12-28T13:56:50,860 Analyzing links from page https://pypi.org/simple/doom-py/ 2020-12-28T13:56:50,860 Found link https://files.pythonhosted.org/packages/6c/64/93ad178a2f85f35f7649e84495d4f3f77c666962b4ed1cf584ba5751a941/doom-py-0.0.1.tar.gz#sha256=940b22f08788d4b75adc09218df3213da012fa334da228a170282e7acf2f1764 (from https://pypi.org/simple/doom-py/), version: 0.0.1 2020-12-28T13:56:50,860 Found link https://files.pythonhosted.org/packages/9e/99/8e2402e26de931f4c2ef80646a0e4ec3d48de144eac870e8aae715742b68/doom-py-0.0.2.tar.gz#sha256=3b538d19ed887f6ec383daa3ce8bef78a048fc5a9f583ae2ea53900e8d8b2e57 (from https://pypi.org/simple/doom-py/), version: 0.0.2 2020-12-28T13:56:50,860 Found link https://files.pythonhosted.org/packages/8f/5d/0096387954dee7f82ce9d326ab7edb02bac36f69bfc075744b7187f6966c/doom-py-0.0.3.tar.gz#sha256=bdf1e1e48a4807f3f40bb1cf162073f8a6e5a4e623314f18012135e28c67758f (from https://pypi.org/simple/doom-py/), version: 0.0.3 2020-12-28T13:56:50,860 Found link https://files.pythonhosted.org/packages/cc/de/d168d1d7ade8be48943857a6f2ab090ae89aff1f943cf39926501cc94e0e/doom-py-0.0.4.tar.gz#sha256=1b2435d31bd226dd1e85aa8d7eb687766097e6c542fd164a7c8b410744087a43 (from https://pypi.org/simple/doom-py/), version: 0.0.4 2020-12-28T13:56:50,861 Found link https://files.pythonhosted.org/packages/8f/f0/fe0d08051808ca7e793a4e5c07a5016e1618a7dff604136efcb4d0917627/doom-py-0.0.5.tar.gz#sha256=0a4307b095a22b37140a3ecbbd3e664bee758ad6258f9401b633b02424c4c5a8 (from https://pypi.org/simple/doom-py/), version: 0.0.5 2020-12-28T13:56:50,861 Found link https://files.pythonhosted.org/packages/64/7d/38d4df5c6ab3bc3a478f36d4d3b3b49c89bca51c754e41dfe89a43139f38/doom-py-0.0.6.tar.gz#sha256=84c13966935f5a09225b4d1922cc5b0204f705b59f9067ddb63b3d1a61c99849 (from https://pypi.org/simple/doom-py/), version: 0.0.6 2020-12-28T13:56:50,861 Found link https://files.pythonhosted.org/packages/36/40/f5af83b228bbf79a50f2edd2f4eb65351649ccdaa8dd9bbb5ff30b377dfc/doom-py-0.0.7.tar.gz#sha256=10a02786f9cad3120a7d4224102381d641146d92c47d4bb007182a2c0d7fafd7 (from https://pypi.org/simple/doom-py/), version: 0.0.7 2020-12-28T13:56:50,861 Found link https://files.pythonhosted.org/packages/58/62/e66c78fbfebbfb875444cc9909eac16e7e81f1840cabe13e08ca750c4dae/doom-py-0.0.8.tar.gz#sha256=5ba84d75832622ab359733e2510cfa89ebb6f13650d295cccef0dfc4cdc648b5 (from https://pypi.org/simple/doom-py/), version: 0.0.8 2020-12-28T13:56:50,861 Found link https://files.pythonhosted.org/packages/f9/d6/667e8673f486f6b28ad25e461c45f94d7cb07d04f6d47422d4dc2674b8ef/doom-py-0.0.9.tar.gz#sha256=ce2f3b93508376472291c8792b8d334679dff508282842bf021d0b9c59e049c0 (from https://pypi.org/simple/doom-py/), version: 0.0.9 2020-12-28T13:56:50,861 Found link https://files.pythonhosted.org/packages/69/c1/c110dd3ee75c1d3e307d84f1d080e8fa52e6a746a8e2f0702be2cc3fd9ad/doom-py-0.0.10.tar.gz#sha256=fe10a5b939a3ab339e00fbd3ef04fd536eb50f55813e859dc60d32aaecd48a47 (from https://pypi.org/simple/doom-py/), version: 0.0.10 2020-12-28T13:56:50,862 Found link https://files.pythonhosted.org/packages/32/6a/27c546411e56848eda57d6210638ef451a233bfee4d3edccf52de8af42c1/doom-py-0.0.11.tar.gz#sha256=23ce4cc8584c38b7a5080f726b8230840b454d82441c5cc6e1a0f277d85df4b6 (from https://pypi.org/simple/doom-py/), version: 0.0.11 2020-12-28T13:56:50,862 Found link https://files.pythonhosted.org/packages/ff/ba/9abd7b3a7422a56fd9351a830a3046b217c187a9aea9679dc80f601944dd/doom-py-0.0.12.tar.gz#sha256=260462420768862fc7f365283c1cd52ae7a91ef74e66d490dba6b5f259725167 (from https://pypi.org/simple/doom-py/), version: 0.0.12 2020-12-28T13:56:50,862 Found link https://files.pythonhosted.org/packages/6c/db/d8574e9fb0eae35e96da7a5371f262b1da6fab568497c56a8519eea5f125/doom-py-0.0.13.tar.gz#sha256=e05c8a73705967e191d51544b296854299d1f56b7270bdb0f41601d262c9dac9 (from https://pypi.org/simple/doom-py/), version: 0.0.13 2020-12-28T13:56:50,862 Found link https://files.pythonhosted.org/packages/bd/26/3680ad1ac28f477a41dcda1ee173c1a4650eecbbacf97adc955a4e770bb9/doom-py-0.0.14.tar.gz#sha256=cc410a749aec6349248a4f5ab4ddb379a2a43eb0fbf06d34004b667976cbb729 (from https://pypi.org/simple/doom-py/), version: 0.0.14 2020-12-28T13:56:50,862 Found link https://files.pythonhosted.org/packages/5f/a4/0700f83e7c1360e981d63737352e9a05f51c6ef553ddc031c41ff0a4ef58/doom-py-0.0.15.tar.gz#sha256=2878cff5fc4b898040664b4a951ae55aeeaef691e3ef38a44fee85799e9b8e75 (from https://pypi.org/simple/doom-py/), version: 0.0.15 2020-12-28T13:56:50,864 Given no hashes to check 15 links for project 'doom-py': discarding no candidates 2020-12-28T13:56:50,864 Using version 0.0.15 (newest of versions: 0.0.1, 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.0.14, 0.0.15) 2020-12-28T13:56:50,865 Collecting doom_py 2020-12-28T13:56:50,866 Created temporary directory: /tmp/pip-unpack-4b6jju12 2020-12-28T13:56:50,883 Using cached https://files.pythonhosted.org/packages/5f/a4/0700f83e7c1360e981d63737352e9a05f51c6ef553ddc031c41ff0a4ef58/doom-py-0.0.15.tar.gz 2020-12-28T13:56:51,629 Added doom_py from https://files.pythonhosted.org/packages/5f/a4/0700f83e7c1360e981d63737352e9a05f51c6ef553ddc031c41ff0a4ef58/doom-py-0.0.15.tar.gz#sha256=2878cff5fc4b898040664b4a951ae55aeeaef691e3ef38a44fee85799e9b8e75 to build tracker '/tmp/pip-req-tracker-1o5ngo8p' 2020-12-28T13:56:51,630 Running setup.py (path:/tmp/pip-install-cy882_jk/doom-py/setup.py) egg_info for package doom-py 2020-12-28T13:56:51,630 Running command python setup.py egg_info 2020-12-28T13:56:52,029 running egg_info 2020-12-28T13:56:52,030 creating /tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info 2020-12-28T13:56:52,030 writing /tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/PKG-INFO 2020-12-28T13:56:52,031 writing dependency_links to /tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/dependency_links.txt 2020-12-28T13:56:52,031 writing requirements to /tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/requires.txt 2020-12-28T13:56:52,031 writing top-level names to /tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/top_level.txt 2020-12-28T13:56:52,032 writing manifest file '/tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/SOURCES.txt' 2020-12-28T13:56:52,074 reading manifest file '/tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/SOURCES.txt' 2020-12-28T13:56:52,074 reading manifest template 'MANIFEST.in' 2020-12-28T13:56:52,152 writing manifest file '/tmp/pip-install-cy882_jk/doom-py/pip-egg-info/doom_py.egg-info/SOURCES.txt' 2020-12-28T13:56:52,173 Source in /tmp/pip-install-cy882_jk/doom-py has version 0.0.15, which satisfies requirement doom_py from https://files.pythonhosted.org/packages/5f/a4/0700f83e7c1360e981d63737352e9a05f51c6ef553ddc031c41ff0a4ef58/doom-py-0.0.15.tar.gz#sha256=2878cff5fc4b898040664b4a951ae55aeeaef691e3ef38a44fee85799e9b8e75 2020-12-28T13:56:52,174 Removed doom_py from https://files.pythonhosted.org/packages/5f/a4/0700f83e7c1360e981d63737352e9a05f51c6ef553ddc031c41ff0a4ef58/doom-py-0.0.15.tar.gz#sha256=2878cff5fc4b898040664b4a951ae55aeeaef691e3ef38a44fee85799e9b8e75 from build tracker '/tmp/pip-req-tracker-1o5ngo8p' 2020-12-28T13:56:52,241 Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from doom_py) (1.19.4) 2020-12-28T13:56:52,243 Building wheels for collected packages: doom-py 2020-12-28T13:56:52,243 Created temporary directory: /tmp/pip-wheel-ovjbcal2 2020-12-28T13:56:52,244 Destination directory: /tmp/pip-wheel-ovjbcal2 2020-12-28T13:56:52,244 Running command /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"'; file='"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ovjbcal2 --python-tag cp36 2020-12-28T13:56:52,665 running bdist_wheel 2020-12-28T13:56:52,714 running build 2020-12-28T13:56:52,864 -- The C compiler identification is GNU 7.5.0 2020-12-28T13:56:52,977 -- The CXX compiler identification is GNU 7.5.0 2020-12-28T13:56:52,993 -- Check for working C compiler: /usr/bin/cc 2020-12-28T13:56:53,077 -- Check for working C compiler: /usr/bin/cc -- works 2020-12-28T13:56:53,079 -- Detecting C compiler ABI info 2020-12-28T13:56:53,161 -- Detecting C compiler ABI info - done 2020-12-28T13:56:53,191 -- Detecting C compile features 2020-12-28T13:56:53,456 -- Detecting C compile features - done 2020-12-28T13:56:53,468 -- Check for working CXX compiler: /usr/bin/c++ 2020-12-28T13:56:53,561 -- Check for working CXX compiler: /usr/bin/c++ -- works 2020-12-28T13:56:53,563 -- Detecting CXX compiler ABI info 2020-12-28T13:56:53,673 -- Detecting CXX compiler ABI info - done 2020-12-28T13:56:53,701 -- Detecting CXX compile features 2020-12-28T13:56:54,186 -- Detecting CXX compile features - done 2020-12-28T13:56:54,271 -- Looking for pthread.h 2020-12-28T13:56:54,359 -- Looking for pthread.h - found 2020-12-28T13:56:54,360 -- Looking for pthread_create 2020-12-28T13:56:54,446 -- Looking for pthread_create - not found 2020-12-28T13:56:54,447 -- Looking for pthread_create in pthreads 2020-12-28T13:56:54,518 -- Looking for pthread_create in pthreads - not found 2020-12-28T13:56:54,519 -- Looking for pthread_create in pthread 2020-12-28T13:56:54,602 -- Looking for pthread_create in pthread - found 2020-12-28T13:56:54,606 -- Found Threads: TRUE 2020-12-28T13:56:54,697 -- Boost version: 1.65.1 2020-12-28T13:56:54,697 -- Found the following Boost libraries: 2020-12-28T13:56:54,697 -- filesystem 2020-12-28T13:56:54,697 -- thread 2020-12-28T13:56:54,697 -- system 2020-12-28T13:56:54,698 -- date_time 2020-12-28T13:56:54,698 -- chrono 2020-12-28T13:56:54,698 -- regex 2020-12-28T13:56:54,698 -- iostreams 2020-12-28T13:56:54,699 -- atomic 2020-12-28T13:56:54,710 CMake Deprecation Warning at cmake_modules/CreateLaunchers.cmake:48 (cmake_policy): 2020-12-28T13:56:54,711 The OLD behavior for policy CMP0026 will be removed from a future version 2020-12-28T13:56:54,711 of CMake.

    2020-12-28T13:56:54,711 The cmake-policies(7) manual explains that the OLD behaviors of all 2020-12-28T13:56:54,711 policies are deprecated and that a policy should be set to OLD only under 2020-12-28T13:56:54,711 specific short-term circumstances. Projects should be ported to the NEW 2020-12-28T13:56:54,711 behavior and not rely on setting a policy to OLD. 2020-12-28T13:56:54,711 Call Stack (most recent call first): 2020-12-28T13:56:54,712 src/vizdoom/CMakeLists.txt:13 (include)

    2020-12-28T13:56:54,722 -- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version "1.0.6") 2020-12-28T13:56:54,725 -- Looking for BZ2_bzCompressInit 2020-12-28T13:56:54,817 -- Looking for BZ2_bzCompressInit - found 2020-12-28T13:56:54,832 -- Found JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (found version "80") 2020-12-28T13:56:54,848 -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11") 2020-12-28T13:56:54,851 -- Could NOT find GME (missing: GME_LIBRARIES GME_INCLUDE_DIR) 2020-12-28T13:56:54,852 -- Using system zlib 2020-12-28T13:56:54,852 -- Using system jpeg library 2020-12-28T13:56:54,852 -- Using system bzip2 library 2020-12-28T13:56:54,852 -- Using internal gme library 2020-12-28T13:56:54,866 -- Performing Test HAVE_NO_ARRAY_BOUNDS 2020-12-28T13:56:54,963 -- Performing Test HAVE_NO_ARRAY_BOUNDS - Success 2020-12-28T13:56:54,975 -- Performing Test __LIBGME_TEST_VISIBILITY 2020-12-28T13:56:55,078 -- Performing Test __LIBGME_TEST_VISIBILITY - Success 2020-12-28T13:56:55,096 -- Looking for strdup 2020-12-28T13:56:55,185 -- Looking for strdup - found 2020-12-28T13:56:55,185 -- Looking for strndup 2020-12-28T13:56:55,273 -- Looking for strndup - found 2020-12-28T13:56:55,274 -- Looking for sys/types.h 2020-12-28T13:56:55,361 -- Looking for sys/types.h - found 2020-12-28T13:56:55,362 -- Looking for stdint.h 2020-12-28T13:56:55,447 -- Looking for stdint.h - found 2020-12-28T13:56:55,448 -- Looking for stddef.h 2020-12-28T13:56:55,530 -- Looking for stddef.h - found 2020-12-28T13:56:55,531 -- Check size of char 2020-12-28T13:56:55,616 -- Check size of char - done 2020-12-28T13:56:55,618 -- Check size of short 2020-12-28T13:56:55,706 -- Check size of short - done 2020-12-28T13:56:55,708 -- Check size of int 2020-12-28T13:56:55,792 -- Check size of int - done 2020-12-28T13:56:55,793 -- Check size of long 2020-12-28T13:56:55,882 -- Check size of long - done 2020-12-28T13:56:55,886 -- /usr/include 2020-12-28T13:56:55,886 -- /usr/include 2020-12-28T13:56:55,886 -- /tmp/pip-install-cy882_jk/doom-py/doom_py/src/vizdoom/lzma/C 2020-12-28T13:56:55,889 -- Looking for itoa 2020-12-28T13:56:55,973 -- Looking for itoa - not found 2020-12-28T13:56:55,985 -- Performing Test DUMB_CAN_USE_SSE 2020-12-28T13:56:56,080 -- Performing Test DUMB_CAN_USE_SSE - Success 2020-12-28T13:56:56,097 -- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1") 2020-12-28T13:56:57,244 -- Checking for module 'gtk+-2.0' 2020-12-28T13:56:57,249 -- No package 'gtk+-2.0' found 2020-12-28T13:56:57,263 CMake Error at /usr/local/lib/python2.7/dist-packages/cmake/data/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:137 (message): 2020-12-28T13:56:57,263 Could NOT find SDL2 (missing: SDL2_LIBRARY SDL2_INCLUDE_DIR) 2020-12-28T13:56:57,264 Call Stack (most recent call first): 2020-12-28T13:56:57,264 /usr/local/lib/python2.7/dist-packages/cmake/data/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE) 2020-12-28T13:56:57,264 cmake_modules/FindSDL2.cmake:179 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) 2020-12-28T13:56:57,264 src/vizdoom/src/CMakeLists.txt:206 (find_package)

    2020-12-28T13:56:57,270 -- Configuring incomplete, errors occurred! 2020-12-28T13:56:57,270 See also "/tmp/pip-install-cy882_jk/doom-py/doom_py/CMakeFiles/CMakeOutput.log". 2020-12-28T13:56:57,270 See also "/tmp/pip-install-cy882_jk/doom-py/doom_py/CMakeFiles/CMakeError.log". 2020-12-28T13:56:57,280  2020-12-28T13:56:57,280 Could not build doom-py: Command '['cmake', '-DCMAKE_BUILD_TYPE=Release', '-DBUILD_PYTHON=ON', '-DBUILD_JAVA=OFF', '-DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3', '-DPYTHON_LIBRARY=/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m']' returned non-zero exit status 1.. (HINT: are you sure cmake is installed? You might also be missing a library. Try running 'apt-get install -y python-numpy cmake zlib1g-dev libjpeg-dev libboost-all-dev gcc libsdl2-dev wget unzip'

    2020-12-28T13:56:57,281 Traceback (most recent call last): 2020-12-28T13:56:57,281 File "", line 1, in 2020-12-28T13:56:57,281 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 88, in 2020-12-28T13:56:57,281 include_package_data=True, 2020-12-28T13:56:57,281 File "/usr/local/lib/python3.6/dist-packages/setuptools/init.py", line 153, in setup 2020-12-28T13:56:57,281 return distutils.core.setup(**attrs) 2020-12-28T13:56:57,281 File "/usr/lib/python3.6/distutils/core.py", line 148, in setup 2020-12-28T13:56:57,282 dist.run_commands() 2020-12-28T13:56:57,282 File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands 2020-12-28T13:56:57,282 self.run_command(cmd) 2020-12-28T13:56:57,282 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command 2020-12-28T13:56:57,282 cmd_obj.run() 2020-12-28T13:56:57,282 File "/usr/local/lib/python3.6/dist-packages/wheel/bdist_wheel.py", line 299, in run 2020-12-28T13:56:57,282 self.run_command('build') 2020-12-28T13:56:57,282 File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command 2020-12-28T13:56:57,283 self.distribution.run_command(command) 2020-12-28T13:56:57,283 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command 2020-12-28T13:56:57,283 cmd_obj.run() 2020-12-28T13:56:57,283 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 63, in run 2020-12-28T13:56:57,283 build_func() 2020-12-28T13:56:57,283 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 41, in build_linux 2020-12-28T13:56:57,283 build_common('so') 2020-12-28T13:56:57,284 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 28, in build_common 2020-12-28T13:56:57,284 subprocess.check_call(['cmake', '-DCMAKE_BUILD_TYPE=Release', '-DBUILD_PYTHON=ON', '-DBUILD_JAVA=OFF', '-DPYTHON_EXECUTABLE:FILEPATH={}'.format(sys.executable)] + cmake_arg_list, cwd='doom_py') 2020-12-28T13:56:57,284 File "/usr/lib/python3.6/subprocess.py", line 311, in check_call 2020-12-28T13:56:57,284 raise CalledProcessError(retcode, cmd) 2020-12-28T13:56:57,284 subprocess.CalledProcessError: Command '['cmake', '-DCMAKE_BUILD_TYPE=Release', '-DBUILD_PYTHON=ON', '-DBUILD_JAVA=OFF', '-DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3', '-DPYTHON_LIBRARY=/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m']' returned non-zero exit status 1. 2020-12-28T13:56:57,308 ERROR: Failed building wheel for doom-py 2020-12-28T13:56:57,309 Running setup.py clean for doom-py 2020-12-28T13:56:57,309 Running command /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"'; file='"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' clean --all 2020-12-28T13:56:57,749 running clean 2020-12-28T13:56:57,750 'build/lib' does not exist -- can't clean it 2020-12-28T13:56:57,750 'build/bdist.linux-x86_64' does not exist -- can't clean it 2020-12-28T13:56:57,750 'build/scripts-3.6' does not exist -- can't clean it 2020-12-28T13:56:57,767 Failed to build doom-py 2020-12-28T13:56:59,644 Installing collected packages: doom-py 2020-12-28T13:56:59,645 Created temporary directory: /tmp/pip-record-po0y591g 2020-12-28T13:56:59,645 Running command /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"'; file='"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-po0y591g/install-record.txt --single-version-externally-managed --compile 2020-12-28T13:57:00,069 running install 2020-12-28T13:57:00,070 running build 2020-12-28T13:57:00,206 -- Boost version: 1.65.1 2020-12-28T13:57:00,206 -- Found the following Boost libraries: 2020-12-28T13:57:00,207 -- filesystem 2020-12-28T13:57:00,207 -- thread 2020-12-28T13:57:00,207 -- system 2020-12-28T13:57:00,207 -- date_time 2020-12-28T13:57:00,207 -- chrono 2020-12-28T13:57:00,207 -- regex 2020-12-28T13:57:00,208 -- iostreams 2020-12-28T13:57:00,208 -- atomic 2020-12-28T13:57:00,219 CMake Deprecation Warning at cmake_modules/CreateLaunchers.cmake:48 (cmake_policy): 2020-12-28T13:57:00,220 The OLD behavior for policy CMP0026 will be removed from a future version 2020-12-28T13:57:00,220 of CMake.

    2020-12-28T13:57:00,220 The cmake-policies(7) manual explains that the OLD behaviors of all 2020-12-28T13:57:00,220 policies are deprecated and that a policy should be set to OLD only under 2020-12-28T13:57:00,220 specific short-term circumstances. Projects should be ported to the NEW 2020-12-28T13:57:00,220 behavior and not rely on setting a policy to OLD. 2020-12-28T13:57:00,221 Call Stack (most recent call first): 2020-12-28T13:57:00,221 src/vizdoom/CMakeLists.txt:13 (include)

    2020-12-28T13:57:00,268 -- Could NOT find GME (missing: GME_LIBRARIES GME_INCLUDE_DIR) 2020-12-28T13:57:00,268 -- Using system zlib 2020-12-28T13:57:00,269 -- Using system jpeg library 2020-12-28T13:57:00,269 -- Using system bzip2 library 2020-12-28T13:57:00,269 -- Using internal gme library 2020-12-28T13:57:00,296 -- /usr/include 2020-12-28T13:57:00,297 -- /usr/include 2020-12-28T13:57:00,297 -- /tmp/pip-install-cy882_jk/doom-py/doom_py/src/vizdoom/lzma/C 2020-12-28T13:57:01,469 -- Checking for module 'gtk+-2.0' 2020-12-28T13:57:01,476 -- No package 'gtk+-2.0' found 2020-12-28T13:57:01,491 CMake Error at /usr/local/lib/python2.7/dist-packages/cmake/data/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:137 (message): 2020-12-28T13:57:01,491 Could NOT find SDL2 (missing: SDL2_LIBRARY SDL2_INCLUDE_DIR) 2020-12-28T13:57:01,491 Call Stack (most recent call first): 2020-12-28T13:57:01,491 /usr/local/lib/python2.7/dist-packages/cmake/data/share/cmake-3.12/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE) 2020-12-28T13:57:01,491 cmake_modules/FindSDL2.cmake:179 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) 2020-12-28T13:57:01,491 src/vizdoom/src/CMakeLists.txt:206 (find_package)

    2020-12-28T13:57:01,497 -- Configuring incomplete, errors occurred! 2020-12-28T13:57:01,497 See also "/tmp/pip-install-cy882_jk/doom-py/doom_py/CMakeFiles/CMakeOutput.log". 2020-12-28T13:57:01,497 See also "/tmp/pip-install-cy882_jk/doom-py/doom_py/CMakeFiles/CMakeError.log". 2020-12-28T13:57:01,505  2020-12-28T13:57:01,506 Could not build doom-py: Command '['cmake', '-DCMAKE_BUILD_TYPE=Release', '-DBUILD_PYTHON=ON', '-DBUILD_JAVA=OFF', '-DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3', '-DPYTHON_LIBRARY=/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m']' returned non-zero exit status 1.. (HINT: are you sure cmake is installed? You might also be missing a library. Try running 'apt-get install -y python-numpy cmake zlib1g-dev libjpeg-dev libboost-all-dev gcc libsdl2-dev wget unzip'

    2020-12-28T13:57:01,506 Traceback (most recent call last): 2020-12-28T13:57:01,506 File "", line 1, in 2020-12-28T13:57:01,506 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 88, in 2020-12-28T13:57:01,506 include_package_data=True, 2020-12-28T13:57:01,506 File "/usr/local/lib/python3.6/dist-packages/setuptools/init.py", line 153, in setup 2020-12-28T13:57:01,506 return distutils.core.setup(**attrs) 2020-12-28T13:57:01,507 File "/usr/lib/python3.6/distutils/core.py", line 148, in setup 2020-12-28T13:57:01,507 dist.run_commands() 2020-12-28T13:57:01,507 File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands 2020-12-28T13:57:01,507 self.run_command(cmd) 2020-12-28T13:57:01,507 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command 2020-12-28T13:57:01,507 cmd_obj.run() 2020-12-28T13:57:01,507 File "/usr/local/lib/python3.6/dist-packages/setuptools/command/install.py", line 61, in run 2020-12-28T13:57:01,507 return orig.install.run(self) 2020-12-28T13:57:01,508 File "/usr/lib/python3.6/distutils/command/install.py", line 589, in run 2020-12-28T13:57:01,508 self.run_command('build') 2020-12-28T13:57:01,508 File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command 2020-12-28T13:57:01,508 self.distribution.run_command(command) 2020-12-28T13:57:01,508 File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command 2020-12-28T13:57:01,508 cmd_obj.run() 2020-12-28T13:57:01,508 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 63, in run 2020-12-28T13:57:01,509 build_func() 2020-12-28T13:57:01,509 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 41, in build_linux 2020-12-28T13:57:01,509 build_common('so') 2020-12-28T13:57:01,509 File "/tmp/pip-install-cy882_jk/doom-py/setup.py", line 28, in build_common 2020-12-28T13:57:01,509 subprocess.check_call(['cmake', '-DCMAKE_BUILD_TYPE=Release', '-DBUILD_PYTHON=ON', '-DBUILD_JAVA=OFF', '-DPYTHON_EXECUTABLE:FILEPATH={}'.format(sys.executable)] + cmake_arg_list, cwd='doom_py') 2020-12-28T13:57:01,509 File "/usr/lib/python3.6/subprocess.py", line 311, in check_call 2020-12-28T13:57:01,509 raise CalledProcessError(retcode, cmd) 2020-12-28T13:57:01,509 subprocess.CalledProcessError: Command '['cmake', '-DCMAKE_BUILD_TYPE=Release', '-DBUILD_PYTHON=ON', '-DBUILD_JAVA=OFF', '-DPYTHON_EXECUTABLE:FILEPATH=/usr/bin/python3', '-DPYTHON_LIBRARY=/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6.so', '-DPYTHON_INCLUDE_DIR=/usr/include/python3.6m']' returned non-zero exit status 1. 2020-12-28T13:57:01,529 Cleaning up... 2020-12-28T13:57:01,529 Removing source in /tmp/pip-install-cy882_jk/doom-py 2020-12-28T13:57:01,618 Removed build tracker '/tmp/pip-req-tracker-1o5ngo8p' 2020-12-28T13:57:01,618 ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"'; file='"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-po0y591g/install-record.txt --single-version-externally-managed --compile Check the logs for full command output. 2020-12-28T13:57:01,618 Exception information: 2020-12-28T13:57:01,618 Traceback (most recent call last): 2020-12-28T13:57:01,618 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/cli/base_command.py", line 153, in _main 2020-12-28T13:57:01,618 status = self.run(options, args) 2020-12-28T13:57:01,618 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/commands/install.py", line 455, in run 2020-12-28T13:57:01,618 use_user_site=options.use_user_site, 2020-12-28T13:57:01,618 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/init.py", line 62, in install_given_reqs 2020-12-28T13:57:01,618 **kwargs 2020-12-28T13:57:01,618 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/req/req_install.py", line 888, in install 2020-12-28T13:57:01,618 cwd=self.unpacked_source_directory, 2020-12-28T13:57:01,618 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/subprocess.py", line 275, in runner 2020-12-28T13:57:01,618 spinner=spinner, 2020-12-28T13:57:01,618 File "/usr/local/lib/python3.6/dist-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess 2020-12-28T13:57:01,618 raise InstallationError(exc_msg) 2020-12-28T13:57:01,618 pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"'; file='"'"'/tmp/pip-install-cy882_jk/doom-py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-po0y591g/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

    Could you please update the README with an alternative method or workaround to this?

    opened by Gummygamer 4
  • Action LSTM Implementation

    Action LSTM Implementation

    Very cool work!

    I have a question regarding the action LSTM implementation (https://github.com/nv-tlabs/GameGAN_code/blob/7e2939563afbbd48d85b1f85828967910bd61fd8/simulator_model/dynamics_engine.py#L65): Should it be g_t = tmp[:, 2 * self.hidden_size : 3 * self.hidden_size].tanh() instead?

    Thanks!

    opened by zhefan 1
  • Any google colaboratory page to use?

    Any google colaboratory page to use?

    Hi, I hope you're having a good day! :3

    So recently I've been diving into Stylegan and Magenta and all sorts of neural networks, it's really fascinating! Now I really want to play around with this gan and really want to do a lot of stuff with this, although, I sadly cannot because of the limitations of my Desktop PC. I do have a nvidia graphics card, although I doubt it'll even run cuda because it's about 11-12 years old. Can I please request for a colaboratory page? I do not know if it is even possible to train this neural network on a V100, P100 or even T4, but I would really appreciate it, I mean, they are pretty good server GPU's, it would just take a long while to train.

    Thanks so much for reading and I hope you have a very good day!

    opened by Randy1435 0
  • Error in data generation

    Error in data generation

    Hello authors,

    Thank you for this amazing paper!

    I tried building the docker using https://github.com/zacwellmer/WorldModels, but I got the following error:

    Step 23/46 : RUN python3 -m pip install --no-cache-dir ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
 ---> Running in ebae39a5f72c
Traceback (most recent call last):
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.5/dist-packages/pip/__main__.py", line 21, in <module>
    from pip._internal.cli.main import main as _main
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/cli/main.py", line 60
    sys.stderr.write(f"ERROR: {exc}")
                                   ^
SyntaxError: invalid syntax

    It seems like the docker is built on python 3.5 and as python 3.5 is deprecated, there are some version incompatibilities. Would it be possible to share either Pacman or Vizdoom data?

    Thanks!

    opened by janismdhanbad 0
Owner
null
GitHub
Code accompanying "Dynamic Neural Relational Inference" from CVPR 2020

Code accompanying "Dynamic Neural Relational Inference" This codebase accompanies the paper "Dynamic Neural Relational Inference" from CVPR 2020. This

Colin Graber 48 Dec 23, 2022
Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments (CoRL 2020)

Motion Planner Augmented Reinforcement Learning for Robot Manipulation in Obstructed Environments [Project website] [Paper] This project is a PyTorch

Cognitive Learning for Vision and Robotics (CLVR) lab @ USC 49 Nov 28, 2022
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Felix Wimbauer 494 Jan 6, 2023
Dynamic View Synthesis from Dynamic Monocular Video

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer This repository contains code to compute depth from a

Intelligent Systems Lab Org 2.3k Jan 1, 2023
Dynamic View Synthesis from Dynamic Monocular Video

Dynamic View Synthesis from Dynamic Monocular Video Project Website | Video | Paper Dynamic View Synthesis from Dynamic Monocular Video Chen Gao, Ayus

Chen Gao 139 Dec 28, 2022
Dynamic vae - Dynamic VAE algorithm is used for anomaly detection of battery data

Dynamic VAE frame Automatic feature extraction can be achieved by probability di

null 10 Oct 7, 2022
Source code for CVPR 2020 paper "Learning to Forget for Meta-Learning"

L2F - Learning to Forget for Meta-Learning Sungyong Baik, Seokil Hong, Kyoung Mu Lee Source code for CVPR 2020 paper "Learning to Forget for Meta-Lear

Sungyong Baik 29 May 22, 2022
Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectrum sensing.

Deep-Learning-based-Spectrum-Sensing Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectru

null 10 Dec 14, 2022
Simulate genealogical trees and genomic sequence data using population genetic models

msprime msprime is a population genetics simulator based on tskit. Msprime can simulate random ancestral histories for a sample of individuals (consis

Tskit developers 150 Dec 14, 2022
duralava is a neural network which can simulate a lava lamp in an infinite loop.

duralava duralava is a neural network which can simulate a lava lamp in an infinite loop. Example This is not a real lava lamp but a "fake" one genera

Maximilian Bachl 87 Dec 20, 2022