GitHub repository for "Improving Video Generation for Multi-functional Applications"

Overview

Improving Video Generation for Multi-functional Applications

GitHub repository for "Improving Video Generation for Multi-functional Applications"

Paper Link

For more information please refer to our homepage.

Requirements

  • Tensorflow 1.2.1
  • Python 2.7
  • ffmpeg

Data Format

Videos are stored as JPEGs of vertically stacked frames. Every frame needs to be at least 64x64 pixels; videos contain between 16 and 32 frames. For an example datasets see: http://carlvondrick.com/tinyvideo/#data

Training

python main_train.py

Important Parameters:

  • mode: one of 'generate', 'predict', 'bw2rgb', 'inpaint' depending on weather you want to generate videos, predict future frames, colorize videos or do inpainting.
  • batch_size: Recommended 64, for colorization use 32 for memory issues.
  • root_dir: root directory of dataset
  • index_file: must be in root_dir, containing a list of all training data clips; path relative to root_dir.
  • experiment_name: name of experiment
  • output_every: output loss to stdout and write to tensorboard summary every xx steps.
  • sample_every: generate a visual sample every xx steps.
  • save_model_very: save the model every xx steps.
  • recover_model: if true recover model and continue training
Comments
  • error when run main_sample.py

    error when run main_sample.py

    sorry for bother u.but when i run the inpaint model i get the issue:

    NotFoundError (see above for traceback): Key g_/g_f_bn0/beta/Adam not found in checkpoint
    [[Node: save/RestoreV2_80 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_80/tensor localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_80/tensor_names, save/RestoreV2_80/shape_and_slices)]]
    [[Node: save/RestoreV2_89/_265 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_506_save/Restore V2_89", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0" ]]

    opened by OnlyFlashEobard 0
  • Modernize Python 2 code to get ready for Python 3

    Modernize Python 2 code to get ready for Python 3

    Make the minimal, safe changes required to convert the repo's code to be syntax compatible with both Python 2 and Python 3. There might be other changes required to complete a port to Python 3 but this PR is a minimal, safe first step.

    Run: futurize --stage1 -w **/*.py

    See Stage 1: "safe" fixes http://python-future.org/automatic_conversion.html#stage-1-safe-fixes
    
    opened by cclauss 0
  • Could you help me?

    Could you help me?

    @bernhard2202 @d-acharya @zzhiwu I am very interested in your work, but when I run the program, there is a bug. The bug is located at utils.py->def convert_img(). I want to know what the purpose of this function is? In addition, the content of the bug is List argument'values' to'ConcatV2' Op with length 0 shorter than minimum length 2. How can I fix this bug? thank you very much!

    opened by supyer9 0
  • How to use this?

    How to use this?

    Hey I am very interested in GANs and how they can create images and videos. However I am totally not a coder. I know how to install packages with apt-get and pip, get most dependencies installed but that's about it. It would be really cool if you could include some more instruction on how to use your code. Especially creating the short videos from still images. I have a virtualenv with py 2.7 and tensorflow 1.2.1 but this it does not work. When running improved_video_gan_future.py I get this

    ~/improved-video-gan/model$ python improved_video_gan_future.py 
    Traceback (most recent call last):
      File "improved_video_gan_future.py", line 6, in <module>
        from utils.layers import conv2d, conv3d_transpose, dis_block, linear
    ImportError: No module named layers
    
    

    This is my pip list

    ~/improved-video-gan/model$ pip list
    Package           Version  
    ----------------- ---------
    absl-py           0.2.2    
    astor             0.6.2    
    backports.weakref 1.0rc1   
    Bashutils         0.0.4    
    bleach            1.5.0    
    enum34            1.1.6    
    funcsigs          1.0.2    
    futures           3.2.0    
    gast              0.2.0    
    grpcio            1.13.0   
    html5lib          0.9999999
    layers            0.1.5    
    Markdown          2.6.11   
    mock              2.0.0    
    numpy             1.14.5   
    pbr               4.0.4    
    Pillow            5.2.0    
    pip               10.0.1   
    pkg-resources     0.0.0    
    protobuf          3.6.0    
    PyYAML            3.12     
    setuptools        39.2.0   
    six               1.11.0   
    tensorflow        1.2.1    
    termcolor         1.1.0    
    utils             0.9.0    
    Werkzeug          0.14.1   
    wheel             0.31.1   
    
    opened by Ianmcmill 1
  • Future Prediction

    Future Prediction

    I am trying to reproduce the results of the future prediction using the same exact code provided. However, I reached step 1200 but with no successful results. Please advise.

    opened by BadourAlBahar 0
Owner
Bernhard Kratzwald
currently: Ph.D. student @ ETH Zurich. past: CS Master @ ETHZ, Bachelor @ TU Vienna
Bernhard Kratzwald
This GitHub repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.'

About Repository This repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.' About Code

Arun Verma 1 Nov 9, 2021
The official GitHub repository for the Argoverse 2 dataset.

Argoverse 2 API Official GitHub repository for the Argoverse 2 family of datasets. If you have any questions or run into any problems with either the

Argo AI 156 Dec 23, 2022
Web-interface + rest API for classification and regression (https://jeff1evesque.github.io/machine-learning.docs)

Machine Learning This project provides a web-interface, as well as a programmatic-api for various machine learning algorithms. Supported algorithms: S

Jeff Levesque 252 Dec 11, 2022
Code for: https://berkeleyautomation.github.io/bags/

DeformableRavens Code for the paper Learning to Rearrange Deformable Cables, Fabrics, and Bags with Goal-Conditioned Transporter Networks. Here is the

Daniel Seita 121 Dec 30, 2022
git《Beta R-CNN: Looking into Pedestrian Detection from Another Perspective》(NeurIPS 2020) GitHub:[fig3]

Beta R-CNN: Looking into Pedestrian Detection from Another Perspective This is the pytorch implementation of our paper "[Beta R-CNN: Looking into Pede

null 35 Sep 8, 2021
git《Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction》(ECCV 2020) GitHub:

Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction Code for the ECCV 2020 paper by Yiming Qian and Yasutaka Furukawa Getting

null 37 Dec 4, 2022
git《Commonsense Knowledge Base Completion with Structural and Semantic Context》(AAAI 2020) GitHub: [fig1]

Commonsense Knowledge Base Completion with Structural and Semantic Context Code for the paper Commonsense Knowledge Base Completion with Structural an

AI2 96 Nov 5, 2022
git《Tangent Space Backpropogation for 3D Transformation Groups》(CVPR 2021) GitHub:1]

LieTorch: Tangent Space Backpropagation Introduction The LieTorch library generalizes PyTorch to 3D transformation groups. Just as torch.Tensor is a m

Princeton Vision & Learning Lab 482 Jan 6, 2023
git《Self-Attention Attribution: Interpreting Information Interactions Inside Transformer》(AAAI 2021) GitHub:

Self-Attention Attribution This repository contains the implementation for AAAI-2021 paper Self-Attention Attribution: Interpreting Information Intera

null 60 Dec 29, 2022
git《Investigating Loss Functions for Extreme Super-Resolution》(CVPR 2020) GitHub:

Investigating Loss Functions for Extreme Super-Resolution NTIRE 2020 Perceptual Extreme Super-Resolution Submission. Our method ranked first and secon

Sejong Yang 0 Oct 17, 2022
git《FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding》(CVPR 2021) GitHub: [fig8]

FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding (CVPR 2021) This repo contains the implementation of our state-of-the-art fewshot ob

null 233 Dec 29, 2022
git《Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser》(2021) GitHub: [fig5]

Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser Abstract The success of deep denoisers on real-world colo

Yue Cao 51 Nov 22, 2022
git《Joint Entity and Relation Extraction with Set Prediction Networks》(2020) GitHub:

Joint Entity and Relation Extraction with Set Prediction Networks Source code for Joint Entity and Relation Extraction with Set Prediction Networks. W

null 130 Dec 13, 2022
git《USD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentation》(2020) GitHub: [fig2]

USD-Seg This project is an implement of paper USD-Seg:Learning Universal Shape Dictionary for Realtime Instance Segmentation, based on FCOS detector f

Ruolin Ye 80 Nov 28, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
《LightXML: Transformer with dynamic negative sampling for High-Performance Extreme Multi-label Text Classification》(AAAI 2021) GitHub:

LightXML: Transformer with dynamic negative sampling for High-Performance Extreme Multi-label Text Classification

null 76 Dec 5, 2022
Github project for Attention-guided Temporal Coherent Video Object Matting.

Attention-guided Temporal Coherent Video Object Matting This is the Github project for our paper Attention-guided Temporal Coherent Video Object Matti

null 71 Dec 19, 2022
Official repo for SemanticGAN https://nv-tlabs.github.io/semanticGAN/

SemanticGAN This is the official code for: Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizat

null 151 Dec 28, 2022
Github for the conference paper GLOD-Gaussian Likelihood OOD detector

FOOD - Fast OOD Detector Pytorch implamentation of the confernce peper FOOD arxiv link. Abstract Deep neural networks (DNNs) perform well at classifyi

null 17 Jun 19, 2022