Code repo for "RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network" (Machine Learning and the Physical Sciences workshop in NeurIPS 2021).

Overview

RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network

An official PyTorch implementation of the RBSRICNN network as described in the paper RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network which is published in the Machine Learning and the Physical Sciences workshop at the 35th Conference on Neural Information Processing Systems (NeurIPS), 2021.

Visual examples:


Abstract

Modern digital cameras and smartphones mostly rely on image signal processing (ISP) pipelines to produce realistic colored RGB images. However, compared to DSLR cameras, low-quality images are usually obtained in many portable mobile devices with compact camera sensors due to their physical limitations. The low-quality images have multiple degradations i.e., sub-pixel shift due to camera motion, mosaick patterns due to camera color filter array, low-resolution due to smaller camera sensors, and the rest information are corrupted by the noise. Such degradations limit the performance of current Single Image Super-resolution (SISR) methods in recovering high-resolution (HR) image details from a single low-resolution (LR) image. In this work, we propose a Raw Burst Super-Resolution Iterative Convolutional Neural Network (RBSRICNN) that follows the burst photography pipeline as a whole by a forward (physical) model. The proposed Burst SR scheme solves the problem with classical image regularization, convex optimization, and deep learning techniques, compared to existing black-box data-driven methods. The proposed network produces the final output by an iterative refinement of the intermediate SR estimates. We demonstrate the effectiveness of our proposed approach in quantitative and qualitative experiments that generalize robustly to real LR burst inputs with onl synthetic burst data available for training.

BibTeX

@InProceedings{Umer_2021_ML4PS,
               author = {Muhammad Umer, Rao and Micheloni, Christian},
               title = {RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network},
               booktitle = {Fourth Workshop on Machine Learning and the Physical Sciences (NeurIPS)},
               month = {December},
               year = {2021}
              }

Quick Test

Dependencies

  • Python 3.7 (version >= 3.0)
  • PyTorch >= 1.0 (CUDA version >= 8.0 if installing with CUDA.)
  • Python packages: pip install numpy opencv-python

Test models

  1. Clone this github repository as the following commands:
git clone https://github.com/RaoUmer/RBSRICNN
cd RBSRICNN
cd test_demo_code
  1. Place the Synthetic Raw LR Burst images in the ./test_demo_code/track1_val_set and ./test_demo_code/track1_test_set folders downloaded from the NTIRE21_BURSTSR.
  2. Place the Real Raw LR Burst images in the ./test_demo_code/track2_val_set and ./test_demo_code/track2_test_set folders downloaded from the NTIRE21_BURSTSR.
  3. Run the tests for the synthetic and real Burst SR by the following provided scripts.
python bsricnn_synsr_val.py
python bsricnn_synsr_test.py
python bsricnn_realsr_val.py
python bsricnn_realsr_test.py
  1. The SR results are into their corresponding ./test_demo_code/sr_results_track{1/2}_{val/test}_set folders.

RBSRICNN Architecture

Overall Representative diagram

Quantitative Results

The quantitative SR results (x4 upscale) are shown over the synthetic and real Burst SR testsets with the common evaluation metrics (PSNR/SSIM/LPIPS). The arrows indicate if high↑ or low↓ values are desired.

Visual Results

Visual comparison of our method with other state-of-the-art methods on the x4 super-resolution over the Raw Burst SR benchmarks. For visual comparison on the benchmarks, you can download our results from the Google Drive: RBSRICNN.

Acknowledgement

The training and testing codes are based on ISRResCNet, burst-photography, and NTIRE21_BURSTSR.

You might also like...
Source codes of CenterTrack++ in 2021 ICME Workshop on Big Surveillance Data Processing and Analysis
Source codes of CenterTrack++ in 2021 ICME Workshop on Big Surveillance Data Processing and Analysis

MOT Tracked object bounding box association (CenterTrack++) New association method based on CenterTrack. Two new branches (Tracked Size and IOU) are a

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.

Data Augmentation for Scene Text Recognition (ICCV 2021 Workshop) (Pronounced as "strog") Paper Arxiv Why it matters? Scene Text Recognition (STR) req

The 1st place solution of track2 (Vehicle Re-Identification) in the NVIDIA AI City Challenge at CVPR 2021 Workshop.

AICITY2021_Track2_DMT The 1st place solution of track2 (Vehicle Re-Identification) in the NVIDIA AI City Challenge at CVPR 2021 Workshop. Introduction

The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Sign Language Translation with Transformers (COLING'2020, ECCV'20 SLRTP Workshop)

transformer-slt This repository gathers data and code supporting the experiments in the paper Better Sign Language Translation with STMC-Transformer.

Pytorch implementation of winner from VQA Chllange Workshop in CVPR'17
Pytorch implementation of winner from VQA Chllange Workshop in CVPR'17

2017 VQA Challenge Winner (CVPR'17 Workshop) pytorch implementation of Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challeng

Workshop Materials Delivered on 28/02/2022

intro-to-cnn-p1 Repo for hosting workshop materials delivered on 28/02/2022 Questions you will answer in this workshop Learning Objectives What are co

Code for MarioNette: Self-Supervised Sprite Learning, in NeurIPS 2021
Code for MarioNette: Self-Supervised Sprite Learning, in NeurIPS 2021

MarioNette | Webpage | Paper | Video MarioNette: Self-Supervised Sprite Learning Dmitriy Smirnov, Michaël Gharbi, Matthew Fisher, Vitor Guizilini, Ale

Owner
Rao Muhammad Umer
Computer Vision & Machine Learning Practitioner
Rao Muhammad Umer
Guiding evolutionary strategies by (inaccurate) differentiable robot simulators @ NeurIPS, 4th Robot Learning Workshop

Guiding Evolutionary Strategies by Differentiable Robot Simulators In recent years, Evolutionary Strategies were actively explored in robotic tasks fo

Vladislav Kurenkov 4 Dec 14, 2021
Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

emmanuel 46 Aug 17, 2022
A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

An Introduction to Deep Learning for the Physical Layer An usable PyTorch implementation of the noisy autoencoder infrastructure in the paper "An Intr

Gram.AI 120 Jul 22, 2022
FCA: Learning a 3D Full-coverage Vehicle Camouflage for Multi-view Physical Adversarial Attack

FCA: Learning a 3D Full-coverage Vehicle Camouflage for Multi-view Physical Adversarial Attack Case study of the FCA. The code can be find in FCA. Cas

IDRL 18 Jul 16, 2022
This repo includes our code for evaluating and improving transferability in domain generalization (NeurIPS 2021)

Transferability for domain generalization This repo is for evaluating and improving transferability in domain generalization (NeurIPS 2021), based on

gordon 8 Aug 20, 2022
Code for "ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on", accepted at WACV 2021 Generation of Human Behavior Workshop.

ShineOn: Illuminating Design Choices for Practical Video-based Virtual Clothing Try-on [ Paper ] [ Project Page ] This repository contains the code fo

Andrew Jong 93 Sep 20, 2022
Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

Unified-EPT Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation. Installation Linux, CUDA>=10.0,

null 29 Aug 23, 2022
Tracking code for the winner of track 1 in the MMP-Tracking Challenge at ICCV 2021 Workshop.

Tracking Code for the winner of track1 in MMP-Trakcing challenge This repository contains our tracking code for the Multi-camera Multiple People Track

DamoCV 26 Aug 17, 2022
Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation.

Physics-Aware Training (PAT) is a method to train real physical systems with backpropagation. It was introduced in Wright, Logan G. & Onodera, Tatsuhiro et al. (2021)1 to train Physical Neural Networks (PNNs) - neural networks whose building blocks are physical systems.

McMahon Lab 216 Sep 29, 2022
Phy-Q: A Benchmark for Physical Reasoning

Phy-Q: A Benchmark for Physical Reasoning Cheng Xue*, Vimukthini Pinto*, Chathura Gamage* Ekaterina Nikonova, Peng Zhang, Jochen Renz School of Comput

null 27 Jul 29, 2022