Super-SloMo 
PyTorch implementation of "Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation" by Jiang H., Sun D., Jampani V., Yang M., Learned-Miller E. and Kautz J. [Project] [Paper]
Check out our paper "Deep Slow Motion Video Reconstruction with Hybrid Imaging System" published in TPAMI.
Results
Results on UCF101 dataset using the evaluation script provided by paper's author. The get_results_bug_fixed.sh script was used. It uses motions masks when calculating PSNR, SSIM and IE.
| Method | PSNR | SSIM | IE |
|---|---|---|---|
| DVF | 29.37 | 0.861 | 16.37 |
| SepConv - L_1 | 30.18 | 0.875 | 15.54 |
| SepConv - L_F | 30.03 | 0.869 | 15.78 |
| SuperSloMo_Adobe240fps | 29.80 | 0.870 | 15.68 |
| pretrained mine | 29.77 | 0.874 | 15.58 |
| SuperSloMo | 30.22 | 0.880 | 15.18 |
Prerequisites
This codebase was developed and tested with pytorch 0.4.1 and CUDA 9.2 and Python 3.6. Install:
For GPU, run
conda install pytorch=0.4.1 cuda92 torchvision==0.2.0 -c pytorch
For CPU, run
conda install pytorch-cpu=0.4.1 torchvision-cpu==0.2.0 cpuonly -c pytorch
- TensorboardX for training visualization
- tensorflow for tensorboard
- matplotlib for training graph in notebook.
- tqdm for progress bar in video_to_slomo.py
- numpy
Training
Preparing training data
In order to train the model using the provided code, the data needs to be formatted in a certain manner. The create_dataset.py script uses ffmpeg to extract frames from videos.
Adobe240fps
For adobe240fps, download the dataset, unzip it and then run the following command
python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset --dataset adobe240fps
Custom
For custom dataset, run the following command
python data\create_dataset.py --ffmpeg_dir path\to\folder\containing\ffmpeg --videos_folder path\to\adobe240fps\videoFolder --dataset_folder path\to\dataset
The default train-test split is 90-10. You can change that using command line argument --train_test_split.
Run the following commmand for help / more info
python data\create_dataset.py --h
Training
In the train.ipynb, set the parameters (dataset path, checkpoint directory, etc.) and run all the cells.
or to train from terminal, run:
python train.py --dataset_root path\to\dataset --checkpoint_dir path\to\save\checkpoints
Run the following commmand for help / more options like continue from checkpoint, progress frequency etc.
python train.py --h
Tensorboard
To get visualization of the training, you can run tensorboard from the project directory using the command:
tensorboard --logdir log --port 6007
and then go to https://localhost:6007.
Evaluation
Pretrained model
You can download the pretrained model trained on adobe240fps dataset here.
Video Converter
You can convert any video to a slomo or high fps video (or both) using video_to_slomo.py. Use the command
# Windows
python video_to_slomo.py --ffmpeg path\to\folder\containing\ffmpeg --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mkv
# Linux
python video_to_slomo.py --video path\to\video.mp4 --sf N --checkpoint path\to\checkpoint.ckpt --fps M --output path\to\output.mkv
If you want to convert a video from 30fps to 90fps set fps to 90 and sf to 3 (to get 3x frames than the original video).
Run the following commmand for help / more info
python video_to_slomo.py --h
You can also use eval.py if you do not want to use ffmpeg. You will instead need to install opencv-python using pip for video IO. A sample usage would be:
python eval.py data/input.mp4 --checkpoint=data/SuperSloMo.ckpt --output=data/output.mp4 --scale=4
Use python eval.py --help for more details
More info TBA
References:
Parts of the code is based on TheFairBear/Super-SlowMo
86 Dec 2, 2022
26 Dec 2, 2022
94 Nov 22, 2022
11 May 19, 2022
789 Jan 4, 2023
114 Nov 26, 2022
192 Dec 12, 2022
196 Nov 27, 2022
436 Jan 9, 2023
2.5k Jan 2, 2023
62 Nov 28, 2022
2 Nov 15, 2021
55 Nov 24, 2022
141 Dec 30, 2022
151 Dec 30, 2022
95 Oct 24, 2022
71 Dec 30, 2022
165 Dec 17, 2022