PyTorch Implementation of "Light Field Image Super-Resolution with Transformers"

Related tags

Deep Learning LFT
Overview

LFT

PyTorch implementation of "Light Field Image Super-Resolution with Transformers", arXiv 2021. [pdf].

Contributions:

  • We make the first attempt to adapt Transformers to LF image processing, and propose a Transformer-based network for LF image SR.
  • We propose a novel paradigm (i.e., angular and spatial Transformers) to incorporate angular and spatial information in an LF.
  • With a small model size and low computational cost, our LFT achieves superior SR performance than other state-of-the-art methods.

Codes and Models:

Requirement

  • PyTorch 1.3.0, torchvision 0.4.1. The code is tested with python=3.6, cuda=9.0.
  • Matlab (For training/test data generation and performance evaluation)

Datasets

We used the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for both training and test. Please first download our dataset via Baidu Drive (key:7nzy) or OneDrive, and place the 5 datasets to the folder ./datasets/.

Train

  • Run Generate_Data_for_Training.m to generate training data. The generated data will be saved in ./data_for_train/ (SR_5x5_2x, SR_5x5_4x).
  • Run train.py to perform network training. Example for training LFT on 5x5 angular resolution for 4x/2xSR: 
    $ python train.py --model_name LFT --angRes 5 --scale_factor 4 --batch_size 4
$ python train.py --model_name LFT --angRes 5 --scale_factor 2 --batch_size 8
  • Checkpoint will be saved to ./log/.

Test

  • Run Generate_Data_for_Test.m to generate test data. The generated data will be saved in ./data_for_test/ (SR_5x5_2x, SR_5x5_4x).
  • Run test.py to perform network inference. Example for test LFT on 5x5 angular resolution for 4x/2xSR: 
    python test.py --model_name LFT --angRes 5 --scale_factor 4 \ 
--use_pre_pth True --path_pre_pth './pth/LFT_5x5_4x_epoch_50_model.pth

python test.py --model_name LFT --angRes 5 --scale_factor 2 \ 
--use_pre_pth True --path_pre_pth './pth/LFT_5x5_2x_epoch_50_model.pth
  • The PSNR and SSIM values of each dataset will be saved to ./log/.

Results:

  • Quantitative Results

  • Efficiency

  • Visual Comparisons

  • Angular Consistency

  • Spatial-Aware Angular Modeling


Citiation

If you find this work helpful, please consider citing:

@Article{LFT,
    author    = {Liang, Zhengyu and Wang, Yingqian and Wang, Longguang and Yang, Jungang and Zhou, Shilin},
    title     = {Light Field Image Super-Resolution with Transformers},
    journal   = {arXiv preprint},
    month     = {August},
    year      = {2021},   
}

Contact

Any question regarding this work can be addressed to [email protected].

Comments
  • About the license for this model

    About the license for this model

    Thank you for sharing your great code. :smiley_cat:

    What is the license for this model? I'd like to cite it to the repository I'm working on if possible, but I want to post the license correctly. https://github.com/PINTO0309/PINTO_model_zoo

    Thank you.

    opened by PINTO0309 2
  • Dataset downloading needs an account that is unable to create with non-Chinese phone

    Dataset downloading needs an account that is unable to create with non-Chinese phone

    Excellent work. It is just that the data downloading pointing to a login window needing an account. Registration to Baidu appears to be challenging using a phone with +44 code (not in the pulldown list).

    opened by xiaohong1 2
  • About the input to the multi-head self-attention in Angular Transformer and Spatial Transformer

    About the input to the multi-head self-attention in Angular Transformer and Spatial Transformer

    To learn the relationship among different angular tokens, I think the shape of the input to MHSA should be (b h w) (a) (c) instead of (a) (b h w) (c) in angular transformer. Similarly, the shape of the input to MHSA should be (b a) (h w) c instead of (h w) (b a) c in spatial transformer.

    opened by HAWKEYE-Group 1
  • Fixed the problem of command breakage during execution

    Fixed the problem of command breakage during execution

    The single space after "\" will break up the shell and cause it to fail to execute.

    $ python test.py --model_name LFT --angRes 5 --scale_factor 4 \ #<--- Here, half-space
--use_pre_pth True --path_pre_pth './pth/LFT_5x5_4x_epoch_50_model.pth'

$ python test.py --model_name LFT --angRes 5 --scale_factor 2 \ #<--- Here, half-space
--use_pre_pth True --path_pre_pth './pth/LFT_5x5_2x_epoch_50_model.pth'

    If we copy the command from the README and execute it as is, the command will be broken in the middle as shown below, so we need to remove the trailing half-width space.

    $ python test.py --model_name LFT --angRes 5 --scale_factor 4 \

$ python test.py --model_name LFT --angRes 5 --scale_factor 2 \
    opened by PINTO0309 0
  • 请问怎么得到rgb的测试结果？

    请问怎么得到rgb的测试结果？

    您好，首先感谢您提供这项工作，在进行训练和测试的时候，数据集都是使用的y通道的数据，在util文件里虽然发现了ycbcr转化为rgb的代码，但是缺少和sr结果等大的cbcr数据无法转换到rgb空间，请问如何得到rgb的测试结果呢？是在rgb空间上训练还是利用其他手段放大cbcr通道的值然后转换呢？谢谢！

    opened by Yueziyu 0
Owner
Squidward
Squidward
GitHub
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
RealFormer-Pytorch Implementation of RealFormer using pytorch

RealFormer-Pytorch Implementation of RealFormer using pytorch. Includes comparison with classical Transformer on image classification task (ViT) wrt C

Simo Ryu 90 Dec 8, 2022
A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

Mixup: Beyond Empirical Risk Minimization in PyTorch This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The co

Harry Yang 121 Dec 17, 2022
A pytorch implementation of Pytorch-Sketch-RNN

Pytorch-Sketch-RNN A pytorch implementation of https://arxiv.org/abs/1704.03477 In order to draw other things than cats, you will find more drawing da

Alexis David Jacq 172 Dec 12, 2022
PyTorch implementation of Advantage async actor-critic Algorithms (A3C) in PyTorch

Advantage async actor-critic Algorithms (A3C) in PyTorch @inproceedings{mnih2016asynchronous, title={Asynchronous methods for deep reinforcement lea

LEI TAI 111 Dec 8, 2022
Pytorch-diffusion - A basic PyTorch implementation of 'Denoising Diffusion Probabilistic Models'

PyTorch implementation of 'Denoising Diffusion Probabilistic Models' This reposi

Arthur Juliani 76 Jan 7, 2023
RetinaNet-PyTorch - A RetinaNet Pytorch Implementation on remote sensing images and has the similar mAP result with RetinaNet in MMdetection

?? RetinaNet Horizontal Detector Based PyTorch This is a horizontal detector Ret

Fang Zhonghao 13 Nov 19, 2022
RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

RETRO - Pytorch (wip) Implementation of RETRO, Deepmind's Retrieval based Attent

Phil Wang 556 Jan 4, 2023
HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

HashNeRF-pytorch Instant-NGP recently introduced a Multi-resolution Hash Encodin

Yash Sanjay Bhalgat 616 Jan 6, 2023
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.

NVIDIA Corporation 6.9k Jan 3, 2023
Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

Mayur 119 Nov 24, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 360 Dec 10, 2022
The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

Ritchie Ng 9.2k Jan 2, 2023
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 359 Jan 5, 2023
A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

Vince 0 Jul 13, 2021
PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

Subin An 8 Nov 21, 2022
PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Amin Rezaei 157 Dec 11, 2022