The official pytorch implemention of the CVPR paper "Temporal Modulation Network for Controllable Space-Time Video Super-Resolution".

Related tags

Deep Learning TMNet
Overview

This is the official PyTorch implementation of TMNet in the CVPR 2021 paper "Temporal Modulation Network for Controllable Space-Time VideoSuper-Resolution"[PDF]. Our TMNet can flexibly interpolate intermediate frames for space-time video super-resolution (STVSR).

Contents

  1. Requirements
  2. Installation
  3. Demo
  4. Training
  5. Testing
  6. Citations

Requirements

Installation

First, make sure your machine has a GPU, which is required for the DCNv2 module.

  1. Clone the TMNet repository.
git clone --recursive https://github.com/CS-GangXu/TMNet.git
  1. Compile the DCNv2:
cd $ROOT/codes/models/modules/DCNv2
bash make.sh
python test.py

Demo (To be uploaded at April 24, 2021 11:59PM (Pacific Time))

Training (To be uploaded at April 24, 2021 11:59PM (Pacific Time))

Testing (To be uploaded at April 24, 2021 11:59PM (Pacific Time))

Citations

If you find the code helpful in your research or work, please cite the following papers.

@InProceedings{xu2021temporal,
  author = {Gang Xu and Jun Xu and Zhen Li and Liang Wang and Xing Sun and Mingming Cheng},
  title = {Temporal Modulation Network for Controllable Space-Time VideoSuper-Resolution},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2021}
}

@InProceedings{xiang2020zooming,
  author = {Xiang, Xiaoyu and Tian, Yapeng and Zhang, Yulun and Fu, Yun and Allebach, Jan P. and Xu, Chenliang},
  title = {Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={3370--3379},
  month = {June},
  year = {2020}
}

@InProceedings{wang2019edvr,
  author    = {Wang, Xintao and Chan, Kelvin C.K. and Yu, Ke and Dong, Chao and Loy, Chen Change},
  title     = {EDVR: Video restoration with enhanced deformable convolutional networks},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  month     = {June},
  year      = {2019},
}

Acknowledgments

Our code is inspired by Zooming-Slow-Mo-CVPR-2020 and EDVR.

Contact

If you have any questions, feel free to E-mail Gang Xu with [email protected].

Issues
  • When I compile, errors come out

    When I compile, errors come out

    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ /usr/bin/nvcc -DWITH_CUDA -I/home/pcl/Yang/TMNet-main/models/modules/DCNv2/src -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/TH -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/THC -I/home/pcl/anaconda3/envs/match/include/python3.6m -c /home/pcl/Yang/TMNet-main/models/modules/DCNv2/src/cuda/dcn_v2_im2col_cuda.cu -o build/temp.linux-x86_64-3.6/home/pcl/Yang/TMNet-main/models/modules/DCNv2/src/cuda/dcn_v2_im2col_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 /usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined

    /usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined

    /home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/c10/util/Half-inl.h(21): error: identifier "__half_as_short" is undefined

    /home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(83): warning: calling a constexpr host function("from_bits") from a host device function("lowest") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

    /home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(84): warning: calling a constexpr host function("from_bits") from a host device function("max") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

    /home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(85): warning: calling a constexpr host function("from_bits") from a host device function("lower_bound") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

    /home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(86): warning: calling a constexpr host function("from_bits") from a host device function("upper_bound") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

    /home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/THC/THCNumerics.cuh(195): error: identifier "__half_as_ushort" is undefined

    4 errors detected in the compilation of "/tmp/tmpxft_00000510_00000000-7_dcn_v2_im2col_cuda.cpp1.ii". error: command '/usr/bin/nvcc' failed with exit status 2

    opened by InstantWindy 6
  • 测试代码问题

    测试代码问题

    你好,感谢您的工作!在运行您的测试代码的时候,我发现checkpoints目录下没有相关的模型,主页上也没有介绍,请问在哪可以获取已经训练好的模型呢?

    opened by silence-moon 6
  • how to load

    how to load "resume_state"?

    Dear guys,I want to loading resume state, May I ask you a question about what should i do? In "config",What is "resume_state" parameter filled in? thank u!!!

    opened by zhengqianisme 2
  • Questions on testing TMNet

    Questions on testing TMNet

    Hi! I'm testing TMNet for interpolating multiple frames on some video datasets and find the results really strange: I aim to interpolate 7 frames between two known frames. When I set the input frame number to 2, the result looks good. However, when I set the input frame number to 4, the results seem to suffer from some unknown noise. I put one example below. The first image is the result of 2 input and the second image is the result of 4 input (Note that in this example the color channel is reversed, but I have corrected it and the problem still exists). I did some further testing and find that when I interpolate less frames(1, 3), the noise disappears. Do you know some possible reasons? Thanks a lot! 3 3

    opened by zychen-ustc 2
  •  No module named 'data.data_sampler'

    No module named 'data.data_sampler'

    Is 'data' a folder to be uploaded or a module can be installed? https://github.com/CS-GangXu/TMNet/blob/992ff0d163760722eecacc7f065bcb85aa3ad06c/codes/train.py#L10 https://github.com/CS-GangXu/TMNet/blob/992ff0d163760722eecacc7f065bcb85aa3ad06c/codes/train.py#L14 https://github.com/CS-GangXu/TMNet/blob/992ff0d163760722eecacc7f065bcb85aa3ad06c/codes/train.py#L18

    opened by Le2Hu 1
  • Gradient updating problem in training

    Gradient updating problem in training

    您好,在训练模型的时候,出现STVSR的out全为nan,是否考虑是梯度更新出现问题,或者是其他问题?

    opened by zhengqianisme 1
  • 提供训练过程的log文件对比参考

    提供训练过程的log文件对比参考

    null

    opened by hhhhhumengshun 1
  • 关于test_multiple_frames.py出现的问题

    关于test_multiple_frames.py出现的问题

    测试时出现的问题 您好,我在测试test_multiple_frames.py文件时出现了上面图中的问题,您能帮我解答下这个具体问题吗?

    opened by zhanglibo852 1
  • bash make.sh

    bash make.sh

    请问cuda版本必须是10.0吗?我的是11.0,修改make.sh后,编译报错

    make.sh: line 1: i#!/usr/bin/env: No such file or directory running build running build_ext building 'ext' extension Emitting ninja build file /mnt/VSR/TMNet/models/modules/DCNv2/build/temp.linux-x86_64-3.6/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] /usr/local/cuda/bin/nvcc -DWITH_CUDA -I/mnt/VSR/TMNet/models/modules/DCNv2/src -I/opt/conda/lib/python3.6/site-packages/torch/include -I/opt/conda/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.6m -c -c /mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.cu -o /mnt/VSR/TMNet/models/modules/DCNv2/build/temp.linux-x86_64-3.6/mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.o -D__CUDA_NO_HALF_OPERATORS_ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="gcc"' '-DPYBIND11_STDLIB="libstdcpp"' '-DPYBIND11_BUILD_ABI="cxxabi1011"' -DTORCH_EXTENSION_NAME=ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14 FAILED: /mnt/VSR/TMNet/models/modules/DCNv2/build/temp.linux-x86_64-3.6/mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.o /usr/local/cuda/bin/nvcc -DWITH_CUDA -I/mnt/VSR/TMNet/models/modules/DCNv2/src -I/opt/conda/lib/python3.6/site-packages/torch/include -I/opt/conda/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/lib/python3.6/site-packages/torch/include/TH -I/opt/conda/lib/python3.6/site-packages/torch/include/THC -I/usr/local/cuda/include -I/opt/conda/include/python3.6m -c -c /mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.cu -o /mnt/VSR/TMNet/models/modules/DCNv2/build/temp.linux-x86_64-3.6/mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.o -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options ''"'"'-fPIC'"'"'' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H '-DPYBIND11_COMPILER_TYPE="_gcc"' '-DPYBIND11_STDLIB="_libstdcpp"' '-DPYBIND11_BUILD_ABI="_cxxabi1011"' -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_75,code=sm_75 -std=c++14 /mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.cu(107): error: identifier "THCState_getCurrentStream" is undefined

    /mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.cu(279): error: identifier "THCState_getCurrentStream" is undefined

    /mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.cu(324): error: identifier "THCudaBlas_Sgemv" is undefined

    3 errors detected in the compilation of "/mnt/VSR/TMNet/models/modules/DCNv2/src/cuda/dcn_v2_cuda.cu". ninja: build stopped: subcommand failed. Traceback (most recent call last): File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1539, in _run_ninja_build env=env) File "/opt/conda/lib/python3.6/subprocess.py", line 438, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last): File "setup.py", line 70, in cmdclass={"build_ext": torch.utils.cpp_extension.BuildExtension}, File "/opt/conda/lib/python3.6/site-packages/setuptools/init.py", line 153, in setup return distutils.core.setup(**attrs) File "/opt/conda/lib/python3.6/distutils/core.py", line 148, in setup dist.run_commands() File "/opt/conda/lib/python3.6/distutils/dist.py", line 955, in run_commands self.run_command(cmd) File "/opt/conda/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/opt/conda/lib/python3.6/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/opt/conda/lib/python3.6/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/opt/conda/lib/python3.6/distutils/dist.py", line 974, in run_command cmd_obj.run() File "/opt/conda/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 79, in run _build_ext.run(self) File "/opt/conda/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run _build_ext.build_ext.run(self) File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 339, in run self.build_extensions() File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 670, in build_extensions build_ext.build_extensions(self) File "/opt/conda/lib/python3.6/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions _build_ext.build_ext.build_extensions(self) File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "/opt/conda/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "/opt/conda/lib/python3.6/distutils/command/build_ext.py", line 533, in build_extension depends=ext.depends) File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 500, in unix_wrap_ninja_compile with_cuda=with_cuda) File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1255, in _write_ninja_file_and_compile_objects error_prefix='Error compiling objects for extension') File "/opt/conda/lib/python3.6/site-packages/torch/utils/cpp_extension.py", line 1555, in _run_ninja_build raise RuntimeError(message) from e

    opened by sunyclj 2
  • 关于预训练模型下载问题

    关于预训练模型下载问题

    您好,下载预训练模型的链接不能进入,还有什么方法能下载吗?

    opened by zhanglibo852 2
  • Distributed training

    Distributed training

    The following errors occurred in distributed training

    image

    opened by InstantWindy 1
Owner
Gang Xu
Gang Xu
Official PyTorch implemention of our paper "Learning to Rectify for Robust Learning with Noisy Labels".

WarPI The official PyTorch implemention of our paper "Learning to Rectify for Robust Learning with Noisy Labels". Run python main.py --corruption_type

Haoliang Sun 1 Dec 6, 2021
The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

null 30 Oct 9, 2021
Super Pix Adv - Offical implemention of Robust Superpixel-Guided Attentional Adversarial Attack (CVPR2020)

Super_Pix_Adv Offical implemention of Robust Superpixel-Guided Attentional Adver

DLight 3 Jan 5, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 61 Jan 11, 2022
Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision https://arxiv.org/abs/2003.00393 Abstract Active learning (AL) aims to min

Denis 27 Oct 27, 2021
Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

Zhengxia Zou 1.2k Jan 13, 2022
Official code of the paper "ReDet: A Rotation-equivariant Detector for Aerial Object Detection" (CVPR 2021)

ReDet: A Rotation-equivariant Detector for Aerial Object Detection ReDet: A Rotation-equivariant Detector for Aerial Object Detection (CVPR2021), Jiam

csuhan 261 Jan 17, 2022
Official code for the paper: Deep Graph Matching under Quadratic Constraint (CVPR 2021)

QC-DGM This is the official PyTorch implementation and models for our CVPR 2021 paper: Deep Graph Matching under Quadratic Constraint. It also contain

Quankai Gao 36 Dec 20, 2021
Official code for the CVPR 2021 paper "How Well Do Self-Supervised Models Transfer?"

How Well Do Self-Supervised Models Transfer? This repository hosts the code for the experiments in the CVPR 2021 paper How Well Do Self-Supervised Mod

Linus Ericsson 117 Jan 15, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 248 Jan 13, 2022
The official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averaging Approach

Graph Optimizer This repo contains the official implementation of our CVPR 2021 paper - Hybrid Rotation Averaging: A Fast and Robust Rotation Averagin

Chenyu 74 Dec 4, 2021
Official source code to CVPR'20 paper, "When2com: Multi-Agent Perception via Communication Graph Grouping"

When2com: Multi-Agent Perception via Communication Graph Grouping This is the PyTorch implementation of our paper: When2com: Multi-Agent Perception vi

null 26 Dec 22, 2021
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 55 Jan 1, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 68 Jan 13, 2022
Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation"

EgoNet Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation". This repo inclu

Shichao Li 87 Jan 12, 2022
Official Implement of CVPR 2021 paper “Cross-Modal Collaborative Representation Learning and a Large-Scale RGBT Benchmark for Crowd Counting”

RGBT Crowd Counting Lingbo Liu, Jiaqi Chen, Hefeng Wu, Guanbin Li, Chenglong Li, Liang Lin. "Cross-Modal Collaborative Representation Learning and a L

null 26 Jan 17, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 26 Nov 27, 2021
The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection .

GCoNet The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection . Trained model Download final_gconet.pth

Qi Fan 36 Nov 8, 2021
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

TUCH This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright License fo

Lea Müller 33 Nov 23, 2021