The official pytorch implemention of the CVPR paper "Temporal Modulation Network for Controllable Space-Time Video Super-Resolution".

cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ /usr/bin/nvcc -DWITH_CUDA -I/home/pcl/Yang/TMNet-main/models/modules/DCNv2/src -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/TH -I/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/THC -I/home/pcl/anaconda3/envs/match/include/python3.6m -c /home/pcl/Yang/TMNet-main/models/modules/DCNv2/src/cuda/dcn_v2_im2col_cuda.cu -o build/temp.linux-x86_64-3.6/home/pcl/Yang/TMNet-main/models/modules/DCNv2/src/cuda/dcn_v2_im2col_cuda.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --compiler-options '-fPIC' -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11 /usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(36): error: identifier "__builtin_ia32_monitorx" is undefined

/usr/lib/gcc/x86_64-linux-gnu/5/include/mwaitxintrin.h(42): error: identifier "__builtin_ia32_mwaitx" is undefined

/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/c10/util/Half-inl.h(21): error: identifier "__half_as_short" is undefined

/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(83): warning: calling a constexpr host function("from_bits") from a host device function("lowest") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(84): warning: calling a constexpr host function("from_bits") from a host device function("max") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(85): warning: calling a constexpr host function("from_bits") from a host device function("lower_bound") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/ATen/cuda/NumericLimits.cuh(86): warning: calling a constexpr host function("from_bits") from a host device function("upper_bound") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.

/home/pcl/anaconda3/envs/match/lib/python3.6/site-packages/torch/include/THC/THCNumerics.cuh(195): error: identifier "__half_as_ushort" is undefined

4 errors detected in the compilation of "/tmp/tmpxft_00000510_00000000-7_dcn_v2_im2col_cuda.cpp1.ii". error: command '/usr/bin/nvcc' failed with exit status 2

Hello, just want to ask during evaluation of fast, medium and slow Vimeo dataset, is it required to change any util code? If yes, which part is necessary to change? I am able to run and evaluate Vid4 dataset, but doesn't seems to know why when running fast/medium/slow vimeo test dataset even I change the glob.glob access, the LR/ HR doesn't seems to be aligning to evaluate. If I keep the original code, the read_image will not read any path, since the original code only have access till , "./datasets/vimeo/LR/00001". Some help will be appreciated! Thank you

Hi, I want to ask question regarding your direction "Please copy the Vimeo7_train_keys.pkl into the folder with higher level. After the aboved operations, we assume that you can get a folder with the following structure". Since there are two Vimeo7_train_keys.pkl file, one generated by GT and another by LR7. I want to ask which one should I be placed in higher level directory?

Best regards, KHC

Hi! I'm testing TMNet for interpolating multiple frames on some video datasets and find the results really strange: I aim to interpolate 7 frames between two known frames. When I set the input frame number to 2, the result looks good. However, when I set the input frame number to 4, the results seem to suffer from some unknown noise. I put one example below. The first image is the result of 2 input and the second image is the result of 4 input (Note that in this example the color channel is reversed, but I have corrected it and the problem still exists). I did some further testing and find that when I interpolate less frames(1, 3), the noise disappears. Do you know some possible reasons? Thanks a lot!

Dear guys,I want to loading resume state, May I ask you a question about what should i do? In "config",What is "resume_state" parameter filled in? thank u!!!

Hello,

I have tried running the model on a custom dataset, but I come across this error:

RuntimeError: CUDA out of memory. Tried to allocate 938.00 MiB (GPU 0; 11.93 GiB total capacity; 1.63 GiB already allocated; 673.69 MiB free; 2.44 GiB reserved in total by PyTorch)

I guess one way to solve this would be to reduce the resolution of my dataset, but is there an alternative way?

Is 'data' a folder to be uploaded or a module can be installed? https://github.com/CS-GangXu/TMNet/blob/992ff0d163760722eecacc7f065bcb85aa3ad06c/codes/train.py#L10 https://github.com/CS-GangXu/TMNet/blob/992ff0d163760722eecacc7f065bcb85aa3ad06c/codes/train.py#L14 https://github.com/CS-GangXu/TMNet/blob/992ff0d163760722eecacc7f065bcb85aa3ad06c/codes/train.py#L18

Hello,

MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

• Video Upscalers Benchmark: Quality Enhancement determines the best upscaling methods for increasing video resolution and improving visual quality.
• Super-Resolution for Video Compression benchmark aims to test Super-Resolution methods on compressed videos and select the best model for each video codec standard.

Your method achieved 15th place in Super-Resolution for Video Compression Benchmark in 'x264 compression' category. We look forward to your future work!

We would be grateful for your feedback on our work.

作者你好，我在训练你的模型时，在第一阶段使用vimeo90k进行单帧训练的时候，超参数的配置按照默认配置，然后在单服务器4张TiTan XP上进行训练，但是仅迭代了5万次，lr下降到3e-4的时候，就出现了DCNv2的offet 超过100的问题，然后我试着去手动调小了lr，但是损失函数几乎没有下降的趋势了，损失值总是维持在差不多9e+4和1e+5这样的范围内

How do we finetune based on previous models?

I assume it's in the config files, particularly here: path: pretrain_model_G: ~ strict_load: true resume_state: ~

Do we simply just specify the directory of the model (pth file) we want to finetune -> pretrain_model_G? What does strict load and resume state do?

When I run 'test_single_frames.py' or 'test_multiple_frames.py', the number of output frames is the same number of input frames. For example, for a specific video with 20 LR frames, the output would be 20 HR frames in the evaluations folder. I would have thought there would be 40 HR frames, similarly to how zooming slow-mo works.

How do I fix this?

Hi,

I'm planning on create a custom dataset for a specific domain which is to be trained on this network and I had a couple of questions of how I should go about this. The plan is to structure the dataset as follows:

customDataset
├── valid
│   ├── HR
│   │   ├── Vid1
│   │   │   ├── 0.png
│   │   │   ├── ...
│   │   │   └── ***.png
│   │   ├── Vid2
│   │   ├── Vid3
│   │   └── Vid4
│   └── LR
│       ├── Vid1
│       │   ├── 0.png
│       │   ├── ...
│       │   └── ***.png
│       ├── Vid2
│       ├── Vid3
│       └── Vid4
└── test
    ├── HR
    │   ├── Vid5
    │   │   ├── 0.png
    │   │   ├── ...
    │   │   └── ***.png
    │   ├── Vid6
    │   ├── Vid7
    │   └── Vid8
    └── LR
        ├── Vid5
        │   ├── 0.png
        │   ├── ...
        │   └── ***.png
        ├── Vid6
        ├── Vid7
        └── Vid8

Questions:

1. Does the HR images need to be a specific size? If they are different sizes, what would I need to change in the configuration files?
2. I would have thought that, given this network is used to solve the STVSR task, there should also be GT (HR) frames corresponding to the non-existent frames which are generated by TMNet. Therefore, the input frames (LR) should be of lower frame rate. But it seems like this isn't the case. Is this correct? So the LR and HR folders should contain the same number of files?