Generating Videos with Scene Dynamics

Related tags

Deep Learning video computer-vision deep-learning generative-adversarial-network
Overview

Generating Videos with Scene Dynamics

This repository contains an implementation of Generating Videos with Scene Dynamics by Carl Vondrick, Hamed Pirsiavash, Antonio Torralba, to appear at NIPS 2016. The model learns to generate tiny videos using adversarial networks.

Example Generations

Below are some selected videos that are generated by our model. These videos are not real; they are hallucinated by a generative video model. While they are not photo-realistic, the motions are fairly reasonable for the scene category they are trained on.

Beach


 Golf
Train Station


 Baby

Training

The code requires a Torch7 installation.

To train a generator for video, see main.lua. This file will construct the networks, start many threads to load data, and train the networks.

For the conditional version, see main_conditional.lua. This is similar to main.lua, except the input to the model is a static image.

To generate videos, see generate.lua. This file will also output intermediate layers, such as the mask and background image, which you can inspect manually.

Data

The data loading is designed assuming videos have been stabilized and flattened into JPEG images. We do this for efficiency. Stabilization is computationally slow and must be done offline, and reading one file per video is more efficient on NFS.

For our stabilization code, see the 'extra' directory. Essentially, this will convert each video into an image of vertically concatenated frames. After doing this, you create a text file listing all the frames, which you pass into the data loader.

Models

You can download our pre-trained models here (1 GB ZIP file).

Notes

The code is based on DCGAN and our starter code in Torch7.

If you find this useful for your research, please consider citing our NIPS paper.

License

MIT

Comments
  • Issues running main.lua

    Issues running main.lua

    Hi

    When I try to run main.lua I get this error message :

    /data/vision/torralba/crossmodal/flickr_videos/scene_extract/lists-full/_b_beach.txt.train : No such file or directory

    Do I have to manually download all the txt file of your project or is there a lua script in the project, which downloads all the data ?

    opened by cryptedp 5
  • Publish datasets as torrent files

    Publish datasets as torrent files

    The article says there's 7 TB of files, they could be published as a set of torrent files that could be community-seeded from different machines.

    It makes sense to split them to amounts individually downloadable to a typical machine (e.g. 7 torrents for 1 tb each)

    opened by wizzard0 2
  • Pre-trained models for condition generation?

    Pre-trained models for condition generation?

    Hi Carl,can you also share the pretrained model for future generation? （This is similar to the request made by @Yuliang-Zou and @17Skye17 before）Before you said this can be done, maybe you forgot to update it ？ This will be very helpful to me.Thank you so much~

    opened by HomeTong 0
  • Confusion about conditional model generation

    Confusion about conditional model generation

    Hi @cvondrick I'm trying to generate a conditional model with main_conditional.lua. The README.md says that a conditional model is trained on a single static image. So, I created a job_list.txt containing a single line referencing a single jpeg file, and ran stabilize_videos_many.py, which extracted a single scene. I then ran main_conditional.lua against the extracted scene. This resulted in 100 iterations through the dataset, and then the program stopped, without saving a model. It looks like main_conditional.lua expects to run 1000 iterations through the dataset, but when I run main_conditional.lua against a single scene, it only run 100 iterations.

    Is there something I should be doing differently? Should I run main_conditional.lua against several identical scenes to mimic a larger dataset? Should I simply set niter to a higher number, or saveIter to a lower one?

    Thanks to you and your colleagues for sharing this great work!

    opened by maxenglander 1
  • Output size of generate.lua

    Output size of generate.lua

    I just ran generate.lua and I get a gif file, which size is 384 pixels and 1408 pixels, which mean that 6 gifs have been stacked horizontally and 22 gifs vertically. I checked the code, and nothing suggests to me, where these stacking is happening. What do I have to change, that the output is only a 64x64 image ?

    opened by cryptedp 1
  • How can I make this work for smaller size inputs?

    How can I make this work for smaller size inputs?

    So I want to run this on 32x32 frames? What all changes will I have to make for this to work? Is there any short way to do it or will I have to hack through the entire code?

    opened by prannayk 0
  • Question of UCF101

    Question of UCF101

    Hi, Carl！ Wonderful job! I just want to know where can I download the file mentioned in your code: /data/vision/torralba/hallucination/UCF101/gan/train.txt

    Best regards.

    opened by ghost 3
Owner
Carl Vondrick
Carl Vondrick
GitHub http://web.mit.edu/vondrick/tinyvideo/
[TIP 2020] Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion Code for Multi-Temporal Scene Classification and Scene Ch

Lixiang Ru 33 Dec 12, 2022
Neural Scene Graphs for Dynamic Scene (CVPR 2021)

Implementation of Neural Scene Graphs, that optimizes multiple radiance fields to represent different objects and a static scene background. Learned representations can be rendered with novel object compositions and views.

null 151 Dec 26, 2022
A weakly-supervised scene graph generation codebase. The implementation of our CVPR2021 paper ``Linguistic Structures as Weak Supervision for Visual Scene Graph Generation''

README.md shall be finished soon. WSSGG 0 Overview 1 Installation 1.1 Faster-RCNN 1.2 Language Parser 1.3 GloVe Embeddings 2 Settings 2.1 VG-GT-Graph

Keren Ye 35 Nov 20, 2022
Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

Cheng Zhang 66 Nov 16, 2022
Automatic number plate recognition using tech: Yolo, OCR, Scene text detection, scene text recognation, flask, torch

Automatic Number Plate Recognition Automatic Number Plate Recognition (ANPR) is the process of reading the characters on the plate with various optica

Meftun AKARSU 52 Dec 22, 2022
Pytorch implementation of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

Make-A-Scene - PyTorch Pytorch implementation (inofficial) of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (https://arxiv.org/

Casual GAN Papers 259 Dec 28, 2022
Weakly Supervised Dense Event Captioning in Videos, i.e. generating multiple sentence descriptions for a video in a weakly-supervised manner.

WSDEC This is the official repo for our NeurIPS paper Weakly Supervised Dense Event Captioning in Videos. Description Repo directories ./: global conf

Melon(Xuguang Duan) 96 Nov 1, 2022
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics

Dataset Cartography Code for the paper Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics at EMNLP 2020. This repository cont

AI2 125 Dec 22, 2022
Code for ECCV 2020 paper "Contacts and Human Dynamics from Monocular Video".

Contact and Human Dynamics from Monocular Video This is the official implementation for the ECCV 2020 spotlight paper by Davis Rempe, Leonidas J. Guib

Davis Rempe 207 Jan 5, 2023
Pytorch code for "State-only Imitation with Transition Dynamics Mismatch" (ICLR 2020)

This repo contains code for our paper State-only Imitation with Transition Dynamics Mismatch published at ICLR 2020. The code heavily uses the RL mach

null 20 Sep 8, 2022
PyTorch Code of "Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spatiotemporal Dynamics"

Memory In Memory Networks It is based on the paper Memory In Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity from Spati

Yang Li 12 May 30, 2022
VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation

VID-Fusion VID-Fusion: Robust Visual-Inertial-Dynamics Odometry for Accurate External Force Estimation Authors: Ziming Ding , Tiankai Yang, Kunyi Zhan

ZJU FAST Lab 86 Nov 18, 2022
Official PyTorch implementation of "Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics".

Physics-aware Difference Graph Networks for Sparsely-Observed Dynamics This repository is the official PyTorch implementation of "Physics-aware Differ

USC-Melady 46 Nov 20, 2022
source code for https://arxiv.org/abs/2005.11248 "Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics"

Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics This work will be published in Nature Biomedical

International Business Machines 71 Nov 15, 2022
PyBullet CartPole and Quadrotor environments—with CasADi symbolic a priori dynamics—for learning-based control and reinforcement learning

safe-control-gym Physics-based CartPole and Quadrotor Gym environments (using PyBullet) with symbolic a priori dynamics (using CasADi) for learning-ba

Dynamic Systems Lab 300 Dec 28, 2022
This is the open-source reference implementation of the SIGGRAPH 2021 paper Intersection-free Rigid Body Dynamics.

Robust, intersection-free, simulations of rigid bodies.

null 93 Dec 23, 2022
Pytorch implementation of CVPR2020 paper “VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation”

VectorNet Re-implementation This is the unofficial pytorch implementation of CVPR2020 paper "VectorNet: Encoding HD Maps and Agent Dynamics from Vecto

null 120 Jan 6, 2023
A python implementation of Physics-informed Spline Learning for nonlinear dynamics discovery

PiSL A python implementation of Physics-informed Spline Learning for nonlinear dynamics discovery. Sun, F., Liu, Y. and Sun, H., 2021. Physics-informe

Fangzheng (Andy) Sun 8 Jul 13, 2022
A Python framework for developing parallelized Computational Fluid Dynamics software to solve the hyperbolic 2D Euler equations on distributed, multi-block structured grids.

pyHype: Computational Fluid Dynamics in Python pyHype is a Python framework for developing parallelized Computational Fluid Dynamics software to solve

Mohamed Khalil 21 Nov 22, 2022