data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"

Last update: Dec 14, 2022

Related tags

Overview

C2F-FWN

data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"
(https://arxiv.org/abs/2012.08976)

News

2020.12.16: Our paper is available on [ArXiv] now!
2020.12.28: Our SoloDance Dataset is available on [google drive] and [baidu pan (extraction code:gle4] now!
2020.12.28: A preview version of our code is now available, which needs further clean-up.

Example Results

motion transfer videos

multi-source appearance attribute editing videos

Full supplementary video: https://youtu.be/THuQN1GXuGI

Prerequisites

Ubuntu
Python 3
NVIDIA GPU (>12GB memory) + CUDA10 cuDNN7
PyTorch 1.0.0

Other Dependencies

DConv (modified from original [DConv])

cd models/dconv
bash make.sh

FlowNet_v2 (directly ported from the original [flownet2] following the steps described in [vid2vid])

cd models/flownet2-pytorch
bash install.sh

Getting Started

It's a preview version of our source code. We will clean it up in the near future.

Notes

Main functions for training and testing can be found in "train_stage1.py", "train_stage2.py", "train_stage2.py", "test_all_stages.py";
Data preprocessings of all the stages can be found in "data" folder;
Model definitions of all the stages can be found in "models" folder;
Training and testing options can be found in "options" folder;
Training and testing scripts can be found in "scripts" folder;
Tool functions can be found in "util" folder.

Data Preparation

Download all the data packages from [google drive] or [baidu pan (extraction code:gle4], and uncompress them. You should create a directory named 'SoloDance' in the root (i.e., 'C2F-FWN') of this project, and then put 'train' and 'test' folders to 'SoloDance' you just created. The structure should look like this:
-C2F-FWN
---SoloDance
------train
------test

Training

1.Train the layout GAN of stage 1:

bash scripts/stage1/train_1.sh

2.Train our C2F-FWN of stage 2:

bash scripts/stage2/train_2_tps_only.sh
bash scripts/stage2/train_2.sh

3.Train the composition GAN of stage 3:

bash scripts/stage3/train_3.sh

Testing all the stages together (separate testing scripts for different stages will be updated in the near future)

bash scripts/full/test_full.sh

Acknowledgement

A large part of the code is borrowed from NVIDIA/vid2vid. Thanks for their wonderful works.

Citation

If you find this project useful for your research, please cite our paper using the following BibTeX entry.

@article{wei2020c2f,
  title={C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer},
  author={Wei, Dongxu and Xu, Xiaowei and Shen, Haibin and Huang, Kejie},
  journal={arXiv preprint arXiv:2012.08976},
  year={2020}
}

Repository for XLM-T, a framework for evaluating multilingual language models on Twitter data

This is the XLM-T repository, which includes data, code and pre-trained multilingual language models for Twitter. XLM-T - A Multilingual Language Mode

112 Dec 27, 2022

Repository for the "Gotta Go Fast When Generating Data with Score-Based Models" paper

Gotta Go Fast When Generating Data with Score-Based Models This repo contains the official implementation for the paper Gotta Go Fast When Generating

89 Nov 9, 2022

This is the research repository for Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition.

Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition This is the research repository for Vid2

26 Dec 24, 2022

NeuralCompression is a Python repository dedicated to research of neural networks that compress data

NeuralCompression is a Python repository dedicated to research of neural networks that compress data. The repository includes tools such as JAX-based entropy coders, image compression models, video compression models, and metrics for image and video evaluation.

297 Jan 6, 2023

Official repository of the paper "A Variational Approximation for Analyzing the Dynamics of Panel Data". Mixed Effect Neural ODE. UAI 2021.

Official repository of the paper (UAI 2021) "A Variational Approximation for Analyzing the Dynamics of Panel Data", Mixed Effect Neural ODE. Panel dat

7 Nov 26, 2022

The repository forked from NVlabs uses our data. (Differentiable rasterization applied to 3D model simplification tasks)

nvdiffmodeling [origin_code] Differentiable rasterization applied to 3D model simplification tasks, as described in the paper: Appearance-Driven Autom

2 Oct 31, 2022

Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA for motion deblurring, image deraining, denoising (Gaussian/real data), and defocus deblurring.

Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan,

906 Dec 30, 2022

Official repository of the paper Privacy-friendly Synthetic Data for the Development of Face Morphing Attack Detectors

SMDD-Synthetic-Face-Morphing-Attack-Detection-Development-dataset Official repository of the paper Privacy-friendly Synthetic Data for the Development

10 Dec 12, 2022

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

1 Oct 26, 2021

Comments

IndexError: list index out of range

Thanks for your great work firstly!

when I run "bash scripts/stage2/train_2.sh", the errors come as below:

2021-02-02T03:36:20.050232142Z Traceback (most recent call last): 2021-02-02T03:36:20.050261277Z File "train_stage2.py", line 148, in 2021-02-02T03:36:20.050276234Z train() 2021-02-02T03:36:20.050280035Z File "train_stage2.py", line 38, in train 2021-02-02T03:36:20.050283718Z for idx, data in enumerate(dataset, start=epoch_iter):
2021-02-02T03:36:20.050287632Z File "/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 637, in next 2021-02-02T03:36:20.050291185Z return self._process_next_batch(batch) 2021-02-02T03:36:20.050307614Z File "/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 658, in _process_next_batch 2021-02-02T03:36:20.050311369Z raise batch.exc_type(batch.exc_msg) 2021-02-02T03:36:20.050314555Z IndexError: Traceback (most recent call last): 2021-02-02T03:36:20.050317753Z File "/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in _worker_loop 2021-02-02T03:36:20.050321226Z samples = collate_fn([dataset[i] for i in batch_indices]) 2021-02-02T03:36:20.050324425Z File "/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 138, in 2021-02-02T03:36:20.050327926Z samples = collate_fn([dataset[i] for i in batch_indices]) 2021-02-02T03:36:20.050331102Z File "/workspace/cpfs-data/C2F-FWN-main-v1/data/data_stage2.py", line 57, in getitem 2021-02-02T03:36:20.050334599Z sparsing_path = self.sparsing_paths[seq_idx][start_idx + i * t_step] 2021-02-02T03:36:20.050337893Z IndexError: list index out of range

I am looking forward to your reply, thanks!

opened by FrankMelon1 1
ModuleNotFoundError: No module named 'models.models'

Thanks for your great work firstly!

when I run "bash scripts/stage1/train_1.sh", the errors come as below:

raceback (most recent call last): File "train_stage1.py", line 10, in from models.models import create_model_parser, create_optimizer_parser, save_models, update_models ModuleNotFoundError: No module named 'models.models' root@notebook-tekxt-jg27l:/workspace/cpfs-data/C2F-FWN-main# bash scripts/stage1/train_1.sh Traceback (most recent call last): File "train_stage1.py", line 10, in from models.models import create_model_parser, create_optimizer_parser, save_models, update_models ModuleNotFoundError: No module named 'models.models'

I am looking forward to your reply, thanks!

opened by FrankMelon1 1
train_stage2 batchsize error

Hello! when I run train_2.sh and set the batchsize as 4, it comes errors as below:

Traceback (most recent call last): File "train_stage2.py", line 154, in train() File "train_stage2.py", line 61, in train fg_tps, fg_dense, lo_tps, lo_dense, flow_tps, flow_dense, flow_totalp, real_input_1, real_input_2, real_SFG, real_SFG_fullp, flow_total_last = ClothWarper(input_TParsing, input_TFG, input_SParsing, input_SFG, input_SFG_full, flow_total_prev_last) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/workspace/cpfs-data/C2F-FWN-main/models/models.py", line 69, in forward outputs = self.model(*inputs, **kwargs, dummy_bs=self.pad_bs) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 141, in forward return self.module(*inputs[0], **kwargs[0]) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/workspace/cpfs-data/C2F-FWN-main/models/model_stage2.py", line 111, in forward warped_fg_tps, warped_fg_dense, warped_lo_tps, warped_lo_dense, flow_tps, flow_dense, flow_total = self.generate_frame_train(net, real_input_1, real_input_2, real_input_TFG, flow_total_prev, start_gpu, is_first_frame) File "/workspace/cpfs-data/C2F-FWN-main/models/model_stage2.py", line 142, in generate_frame_train = net.forward(real_input_1_reshaped, real_input_2_reshaped, real_input_3_reshaped, real_input_tfg_reshaped) File "/workspace/cpfs-data/C2F-FWN-main/models/networks.py", line 474, in forward feature_T_1 = self.model_down_target_1([feature_T_0, input_tlo_0, input_tlo_1]) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/workspace/cpfs-data/C2F-FWN-main/models/dconv/modules/modulated_deform_conv.py", line 173, in forward sample_LO = torch.nn.functional.grid_sample(input_LO, sample_location, mode='bilinear', padding_mode='border') File "/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py", line 2597, in grid_sample return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum) RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [64, 3, 128,96] and grid with sizes [256, 64, 48, 2]

the train_2.sh are shown below:

python train_stage2.py --name clothwarp_256p
--dataroot /workspace/cpfs-data/data/ICCV2021/SoloDance/SoloDance_upload/train --dataset_mode cloth --model cloth --nThreads 16
--input_nc_T_2 4 --input_nc_S_2 3 --input_nc_P_2 10 --ngf 64 --n_downsample_warper 4 --label_nc_2 3 --grid_size 3
--resize_or_crop scaleHeight --loadSize 256 --random_drop_prob 0 --color_aug
--gpu_ids 0 --n_gpus_gen 1 --batchSize 4 --max_frames_per_gpu 12 --display_freq 40 --print_freq 40 --save_latest_freq 1000
--niter 5 --niter_decay 5 --n_scales_temporal 3 --n_frames_D 2
--no_first_img --n_frames_total 12 --max_t_step 4 --tf_log --continue

hope for your replay!

opened by FrankMelon1 0

data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"

Related tags

Overview

C2F-FWN

News

Example Results

Prerequisites

Other Dependencies

DConv (modified from original [DConv])

FlowNet_v2 (directly ported from the original [flownet2] following the steps described in [vid2vid])

Getting Started

Notes

Data Preparation

Training

1.Train the layout GAN of stage 1:

2.Train our C2F-FWN of stage 2:

3.Train the composition GAN of stage 3:

Testing all the stages together (separate testing scripts for different stages will be updated in the near future)

Acknowledgement

Citation

You might also like...

Repository for XLM-T, a framework for evaluating multilingual language models on Twitter data

Repository for the "Gotta Go Fast When Generating Data with Score-Based Models" paper

This is the research repository for Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition.

NeuralCompression is a Python repository dedicated to research of neural networks that compress data

Official repository of the paper "A Variational Approximation for Analyzing the Dynamics of Panel Data". Mixed Effect Neural ODE. UAI 2021.

The repository forked from NVlabs uses our data. (Differentiable rasterization applied to 3D model simplification tasks)

Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA for motion deblurring, image deraining, denoising (Gaussian/real data), and defocus deblurring.

Official repository of the paper Privacy-friendly Synthetic Data for the Development of Face Morphing Attack Detectors

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Comments

IndexError: list index out of range

ModuleNotFoundError: No module named 'models.models'

train_stage2 batchsize error

Owner

EKILI

This repository is related to an Arabic tutorial, within the tutorial we discuss the common data structure and algorithms and their worst and best case for each, then implement the code using Python.

This repository contains the code, data, and models of the paper titled "XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages" published in Findings of the Association for Computational Linguistics: ACL 2021.

This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

This repository holds code and data for our PETS'22 article 'From "Onion Not Found" to Guard Discovery'.

This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" (SPNLP@ACL2022)

This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

Automatically download the cwru data set, and then divide it into training data set and test data set

The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.