PyTorch implementation of ECCV 2020 paper "Foley Music: Learning to Generate Music from Videos "

Last update: Nov 3, 2022

Related tags

Overview

Foley Music: Learning to Generate Music from Videos

This repo holds the code for the framework presented on ECCV 2020.

Foley Music: Learning to Generate Music from Videos Chuang Gan, Deng Huang, Peihao Chen, Joshua B. Tenenbaum, and Antonio Torralba

paper

Usage Guide

Prerequisites

The training and testing in PGCN is reimplemented in PyTorch for the ease of use.

Pytorch 1.4

Other minor Python modules can be installed by running

pip install -r requirements.txt

Data Preparation

Download Datasets

The extracted pose and midi for training and audio generation can be downloaded here and unzip to ./data folder.

The original datasets (including videos) can be found:

URMP: can be downloaded here
MUSIC: can be downloaded here
AtinPiano: proposed by At Your Fingertips: Automatic Piano Fingering Detection. The dataset can be downloaded here

Training

For URMP

CUDA_VISIBLE_DEVICES=6 python train.py -c config/URMP/violin.conf -e exps/urmp-vn

For AtinPiano

CUDA_VISIBLE_DEVICES=6 python train.py -c config/AtinPiano.conf -e exps/atinpiano

For MUSIC

CUDA_VISIBLE_DEVICES=6 python train.py -c config/MUSIC/accordion.conf -e exps/music-accordion

Generating MIDI, sounds and videos

For URMP

VIDEO_PATH=/path/to/video
INSTRUMENT_NAME='Violin'
python test_URMP.py exps/urmp-vn/checkpoint.pth.tar -o exps/urmp-vn/generate -i Violin -v $VIDEO_PATH -i $INSTRUMENT_NAME

For AtinPiano

VIDEO_PATH=/path/to/video
INSTRUMENT_NAME='Acoustic Grand Piano'
python test_AtinPiano_MUSIC.py exps/atinpiano/checkpoint.pth.tar -o exps/atinpiano/generation -v $VIDEO_PATH -i $INSTRUMENT_NAME

For MUSIC

VIDEO_PATH=/path/to/video
INSTRUMENT_NAME='Accordion'
python test_AtinPiano_MUSIC.py exps/music-accordion/checkpoint.pth.tar -o exps/music-accordion/generation -v $VIDEO_PATH -i $INSTRUMENT_NAME

Notes:

Instrument name ($INSTRUMENT_NAME) can be found here
If you do not have the video file or you want to generate MIDI and audio only, you can add -oa flag to skip the generation of video.

Other Info

Citation

Please cite the following paper if you feel our work useful to your research.

@inproceedings{FoleyMusic2020,
  author    = {Chuang Gan and
               Deng Huang and
               Peihao Chen and
               Joshua B. Tenenbaum and
               Antonio Torralba},
  title     = {Foley Music: Learning to Generate Music from Videos},
  booktitle = {ECCV},
  year      = {2020},
}

git《Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction》(ECCV 2020) GitHub:

Learning Pairwise Inter-Plane Relations for Piecewise Planar Reconstruction Code for the ECCV 2020 paper by Yiming Qian and Yasutaka Furukawa Getting

37 Dec 4, 2022

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Motion Capture from Internet Videos Motion Capture from Internet Videos Junting Dong*, Qing Shuai*, Yuanqing Zhang, Xian Liu, Xiaowei Zhou, Hujun Bao

98 Dec 7, 2022

《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

42 Nov 24, 2022

[ECCV 2020] Gradient-Induced Co-Saliency Detection

Gradient-Induced Co-Saliency Detection Zhao Zhang*, Wenda Jin*, Jun Xu, Ming-Ming Cheng ⭐ Project Home » The official repo of the ECCV 2020 paper Grad

35 Nov 25, 2022

Code for Towards Streaming Perception (ECCV 2020) :car:

sAP — Code for Towards Streaming Perception ECCV Best Paper Honorable Mention Award Feb 2021: Announcing the Streaming Perception Challenge (CVPR 2021

85 Dec 22, 2022

Sign Language Translation with Transformers (COLING'2020, ECCV'20 SLRTP Workshop)

transformer-slt This repository gathers data and code supporting the experiments in the paper Better Sign Language Translation with STMC-Transformer.

107 Dec 27, 2022

Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

Progressive Transformers for End-to-End Sign Language Production Source code for "Progressive Transformers for End-to-End Sign Language Production" (B

58 Dec 21, 2022

IAST: Instance Adaptive Self-training for Unsupervised Domain Adaptation (ECCV 2020)

This repo is the official implementation of our paper "Instance Adaptive Self-training for Unsupervised Domain Adaptation". The purpose of this repo is to better communicate with you and respond to your questions. This repo is almost the same with Another-Version, and you can also refer to that version.

84 Dec 12, 2022

Boundary-preserving Mask R-CNN (ECCV 2020)

BMaskR-CNN This code is developed on Detectron2 Boundary-preserving Mask R-CNN ECCV 2020 Tianheng Cheng, Xinggang Wang, Lichao Huang, Wenyu Liu Video

178 Nov 28, 2022

Comments

Erro with torchpie packet

I get the following erro: Traceback (most recent call last): File "/disk/11811901/anaconda3/envs/torchb/python3.7/site-packages/injector/init.py", line 804, in get return self._context[key] KeyError: <class 'torchpie.core.environment.Args'>

and a exception occur when handling this exception packages/torchpie/core/environment/init.py", line 112, in provide_args args = Args() File "/disk/11811901/anaconda3/envs/torch/lib/python3.7/site-packages/torchpie/core/environment/init.py", line 75, in init self.parse_known_args_from(parser) AttributeError: 'Args' object has no attribute 'parse_known_args_from'

I get my torchpie packet by running pip install -r requirements.txt and train the URMP by CUDA_VISIBLE_DEVICES=6 python train.py -c config/URMP/violin.conf -e exps/urmp-vn with torch version 1.4

I check my version of torchpie,it seems that it do not have parse_known_args_from, and I only find few message about this packet from google.

opened by santyelegy 4

cannot install depending in requirements.txt

Tried to install dependency in requirements, ran into the following error.

pip install -r requirements.txt
Collecting git+https://git.dev.tencent.com/SunDoge/[email protected] (from -r requirements.txt (line 1))
  Cloning https://git.dev.tencent.com/SunDoge/torchpie.git (to v0.2) to /tmp/pip-nOruUH-build
fatal: unable to access 'https://git.dev.tencent.com/SunDoge/torchpie.git/': Failed to connect to git.dev.tencent.com port 443: Connection timed out
Command "git clone -q https://git.dev.tencent.com/SunDoge/torchpie.git /tmp/pip-nOruUH-build" failed with error code 128 in None

opened by mr3albert 2

Owner

Chuang Gan

Researcher and Engineer on Deep Learning and Computer Vision

GitHub

PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

HAN PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network" This repository is for HAN introduced in the

140 Nov 23, 2022

Unofficial PyTorch implementation of "RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving" (ECCV 2020)

RTM3D-PyTorch The PyTorch Implementation of the paper: RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving (ECCV 2020

271 Nov 29, 2022