Image restoration with neural networks but without learning.

Overview

Warning! The optimization may not converge on some GPUs. We've personally experienced issues on Tesla V100 and P40 GPUs. When running the code, make sure you get similar results to the paper first. Easiest to check using text inpainting notebook. Try to set double precision mode or turn off cudnn.

Deep image prior

In this repository we provide Jupyter Notebooks to reproduce each figure from the paper:

Deep Image Prior

CVPR 2018

Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky

[paper] [supmat] [project page]

Here we provide hyperparameters and architectures, that were used to generate the figures. Most of them are far from optimal. Do not hesitate to change them and see the effect.

We will expand this README with a list of hyperparameters and options shortly.

Install

Here is the list of libraries you need to install to execute the code:

  • python = 3.6
  • pytorch = 0.4
  • numpy
  • scipy
  • matplotlib
  • scikit-image
  • jupyter

All of them can be installed via conda (anaconda), e.g.

conda install jupyter

or create an conda env with all dependencies via environment file

conda env create -f environment.yml

Docker image

Alternatively, you can use a Docker image that exposes a Jupyter Notebook with all required dependencies. To build this image ensure you have both docker and nvidia-docker installed, then run

nvidia-docker build -t deep-image-prior .

After the build you can start the container as

nvidia-docker run --rm -it --ipc=host -p 8888:8888 deep-image-prior

you will be provided an URL through which you can connect to the Jupyter notebook.

Google Colab

To run it using Google Colab, click here and select the notebook to run. Remember to uncomment the first cell to clone the repository into colab's environment.

Citation

@article{UlyanovVL17,
    author    = {Ulyanov, Dmitry and Vedaldi, Andrea and Lempitsky, Victor},
    title     = {Deep Image Prior},
    journal   = {arXiv:1711.10925},
    year      = {2017}
}
Issues
  • RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2864

    RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2864

    env:

    • pytorch 0.3
    • python 3.5
    • cuda8.0

    inpainting.ipynb got this error at

    Visualize : plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-6-e9e29a4716c2> in <module>()
          1 img_mask_var = np_to_var(img_mask_np).type(dtype)
          2 
    ----> 3 plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);
    
    /data/notebooks/deep-image-prior/utils/common_utils.py in plot_image_grid(images_np, nrow, factor, interpolation)
         70         interpolation: interpolation used in plt.imshow
         71     """
    ---> 72     grid = get_image_grid(images_np, nrow)
         73 
         74     plt.figure(figsize=(len(images_np)+factor,12+factor))
    
    /data/notebooks/deep-image-prior/utils/common_utils.py in get_image_grid(images_np, nrow)
         57     '''Creates a grid from a list of images by concatenating them.'''
         58     images_torch = [torch.from_numpy(x) for x in images_np]
    ---> 59     torch_grid = torchvision.utils.make_grid(images_torch, nrow)
         60 
         61     return torch_grid.numpy()
    
    /usr/local/lib/python3.5/dist-packages/torchvision/utils.py in make_grid(tensor, nrow, padding, normalize, range, scale_each, pad_value)
         33     # if list of tensors, convert to a 4D mini-batch Tensor
         34     if isinstance(tensor, list):
    ---> 35         tensor = torch.stack(tensor, dim=0)
         36 
         37     if tensor.dim() == 2:  # single image H x W
    
    /usr/local/lib/python3.5/dist-packages/torch/functional.py in stack(sequence, dim, out)
         62     inputs = [t.unsqueeze(dim) for t in sequence]
         63     if out is None:
    ---> 64         return torch.cat(inputs, dim)
         65     else:
         66         return torch.cat(inputs, dim, out=out)
    
    RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2864
    
    opened by eromoe 7
  • Cuda Error

    Cuda Error

    I tried to run this with few other images. I am getting this error. Can you please tell me where I am going wrong? The following is the error I am getting.

    RuntimeError Traceback (most recent call last) in () ----> 1 net_input = get_noise(input_depth, INPUT, (imgs['HR_pil'].size[1], imgs['HR_pil'].size[0])).type(dtype).detach() 2 3 NET_TYPE = 'skip' # UNet, ResNet 4 net = get_net(input_depth, 'skip', pad, 5 skip_n33d=128,

    RuntimeError: CUDA error: out of memory

    opened by sanayam 5
  • Replacement for the image of Lena

    Replacement for the image of Lena

    This is not a technical bug, but an important matter you want to address.

    In the selection of example images, you have included the photo known as Lena (or Lenna).

    While I'm guessing you might want to show the behavior of your algorithm on human faces, the choice of the image in this case is quite unfortunate and widely discussed as an unwanted picture in today's image processing sets.

    Inspired by other projects, please consider using one of these images instead:

    Thank you for your help and addressing this report with the most careful attention.

    Have a nice day

    opened by xuv 4
  • import error

    import error

    Hi, @DmitryUlyanov ,

    I got the following import error, which shows it cannot find skip. I think that's path pointing issue. Would you give some suggestion to fix it? THX!

    ModuleNotFoundError                       Traceback (most recent call last)
    <ipython-input-1-9abca6442a47> in <module>()
          7 
          8 import numpy as np
    ----> 9 from models import *
         10 
         11 import torch
    
    /data/code/deep-image-prior/models/__init__.py in <module>()
    ----> 1 from skip import skip
          2 from texture_nets import get_texture_nets
          3 from resnet import ResNet
          4 # from model_pad_ds import Net
          5 from unet import UNet
    
    ModuleNotFoundError: No module named 'skip'
    
    opened by amiltonwong 4
  • Flash, no Flash

    Flash, no Flash

    Do you have the example for the flash, no flash figure? In the paper, it says you used the same parameters as the denoising example. Is this correct?

    Thanks

    opened by nikky4D 3
  • Torch CUDA

    Torch CUDA

    IMO the install section should mention that pytorch needs to be compiled with CUDA support, which is quite fiddly on some macOS versions (needing to downgrade clang to compile at all, since nvcc doesn't like clang 90000).

    opened by turbo 3
  • Images should have 1 or 3 channels

    Images should have 1 or 3 channels

    I'm trying to test with my own image and mask and running into an issue on : plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

    assert (n_channels == 3) or (n_channels == 1), "images should have 1 or 3 channels"

    I've ensured that my image only has 3 channels i.e. RGB, and removed the alpha channel. But I"m guessing maybe this isn't what it's referring to?

    opened by Tetsuo7945 2
  • ValueError: operands could not be broadcast together with shapes

    ValueError: operands could not be broadcast together with shapes

    ValueError Traceback (most recent call last)

    in () 1 img_mask_var = np_to_torch(img_mask_np).type(dtype) 2 ----> 3 plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

    ValueError: operands could not be broadcast together with shapes (4,320,320) (3,320,320)

    I have that strange error, does anyone know how to fix it?

    opened by avmiz 2
  • Error while backtracking in restoration notebook

    Error while backtracking in restoration notebook

    Backtracking condition fails and gives an error.

    Falling back to previous checkpoint.R_masked 11.899794 PSNR 11.899794 
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-16-f44ea9e7035f> in <module>()
         50 # Run
         51 p = get_params(OPT_OVER, net, net_input)
    ---> 52 optimize(OPTIMIZER, p, closure, LR=LR, num_iter=num_iter)
    
    /nfs1/code/aniruddha/deep-image-prior/utils/common_utils.py in optimize(optimizer_type, parameters, closure, LR, num_iter)
        227         for j in range(num_iter):
        228             optimizer.zero_grad()
    --> 229             closure()
        230             optimizer.step()
        231     else:
    
    <ipython-input-16-f44ea9e7035f> in closure()
         25 
         26             for new_param, net_param in zip(last_net, net.parameters()):
    ---> 27                 net_param.copy_(new_param.cuda())
         28 
         29             return total_loss*0
    
    RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.
    

    I could reproduce this multiple times.

    opened by ani0075 2
  • Batch Normalization

    Batch Normalization

    There are a lot of Batch Normalization layers in the models, but the code only uses batch size of 1. What role does the BN play in the model? Have you tried training without BN and how did it perform?

    opened by joeyhng 2
  • Error in forward method

    Error in forward method

    Hi I am debugging your code. But there is problem in forward function. As a matter of fact, I debugged it for hours and could figure out that problem is in concat sections because the size of features extracted in both layers for concatenation haven't match. I am debugging denoising algorithm.

    The error is shown here:

        Traceback (most recent call last):
    
    • File "/home/morteza/Desktop/skin detection/Prior/deep-image-prior-master/main.py", line 500, in
    • optimize(OPTIMIZER, p, closure, LR, num_iter)
      
    • File "/home/morteza/Desktop/skin detection/Prior/deep-image-prior-master/utils/common_utils.py", line 229, in optimize
    • closure()
      
    • File "/home/morteza/Desktop/skin detection/Prior/deep-image-prior-master/main.py", line 483, in closure
    • out = net(net_input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/Desktop/skin detection/Prior/deep-image-prior-master/models/common.py", line 21, in forward
    • inputs.append(module(input))
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward
    • input = module(input)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in call
    • result = self.forward(*input, **kwargs)
      
    • File "/home/morteza/Desktop/skin detection/Prior/deep-image-prior-master/models/common.py", line 23, in forward
    • return torch.cat(inputs, dim=self.dim)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/autograd/variable.py", line 897, in cat
    • return Concat.apply(dim, *iterable)
      
    • File "/home/morteza/.local/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 317, in forward
    • return torch.cat(inputs, dim)
      
    • RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2709

    please guide me. I am struggling to deeply know your method.

    opened by mortezamg63 2
  • What is the skip net used for inpainting?

    What is the skip net used for inpainting?

    Hi. Can you provide some refs or papers of the skip net used for inpainting? I want to know its structure and the difference between skip-net and unet.

    opened by daoyuli 0
  • Error inpainting for CUDA

    Error inpainting for CUDA

    Hi, In inpainting.ipynb the loss is NAN from the beginning. However, if I turn off cuda:

      torch.backends.cudnn.enabled = False 
      torch.backends.cudnn.benchmark = False 
      dtype = torch.FloatTensor 
    

    All the loss is right. I was using RTX 3070 Ti, cuda 9.0.

    Why the loss turns into 'nan' when running on GPU? Please help me with this problem.

    opened by June-phy 0
  • Python package

    Python package

    Hi Dmitry, First of all, great explanations, codes and repo. Do you intend to create a python package with this method? I believe it would be of great use.

    opened by mvpalheta 0
  • Error in exporting final image

    Error in exporting final image

    I was trying to use the inpaint model that yo guys have and the final image is stored in out_np, how do I got this to a picture file that I can download. I tried using cv2, matplotlib, and PIL to accomplish this with no luck. How am I supposed to export out_np as an image?

    opened by throwawayaccer 0
Owner
Dmitry Ulyanov
Co-Founder at in3D, Phd @ Skoltech
Dmitry Ulyanov
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

GarField 85 Jul 13, 2022
Dynamic Attentive Graph Learning for Image Restoration, ICCV2021 [PyTorch Code]

Dynamic Attentive Graph Learning for Image Restoration This repository is for GATIR introduced in the following paper: Chong Mou, Jian Zhang, Zhuoyuan

Jian Zhang 77 Jul 16, 2022
Multi-Stage Progressive Image Restoration

Multi-Stage Progressive Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Sh

Syed Waqas Zamir 766 Aug 9, 2022
(under submission) Bayesian Integration of a Generative Prior for Image Restoration

BIGPrior: Towards Decoupling Learned Prior Hallucination and Data Fidelity in Image Restoration Authors: Majed El Helou, and Sabine Süsstrunk {Note: p

Majed El Helou 21 Apr 17, 2022
HINet: Half Instance Normalization Network for Image Restoration

HINet: Half Instance Normalization Network for Image Restoration Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, Chengpeng Chen Paper: https://arxiv.org

null 276 Aug 9, 2022
EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration

EDPN: Enhanced Deep Pyramid Network for Blurry Image Restoration Ruikang Xu, Zeyu Xiao, Jie Huang, Yueyi Zhang, Zhiwei Xiong. EDPN: Enhanced Deep Pyra

null 68 Aug 2, 2022
Real-ESRGAN aims at developing Practical Algorithms for General Image Restoration.

Real-ESRGAN Colab Demo for Real-ESRGAN . Portable Windows executable file. You can find more information here. Real-ESRGAN aims at developing Practica

Xintao 12.7k Aug 10, 2022
SwinIR: Image Restoration Using Swin Transformer

SwinIR: Image Restoration Using Swin Transformer This repository is the official PyTorch implementation of SwinIR: Image Restoration Using Shifted Win

Jingyun Liang 1.9k Aug 16, 2022
Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR

Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR

Kai Zhang 1.7k Aug 11, 2022
An official repository for Paper "Uformer: A General U-Shaped Transformer for Image Restoration".

Uformer: A General U-Shaped Transformer for Image Restoration Zhendong Wang, Xiaodong Cun, Jianmin Bao and Jianzhuang Liu Paper: https://arxiv.org/abs

Zhendong Wang 418 Aug 13, 2022
Image Restoration Using Swin Transformer for VapourSynth

SwinIR SwinIR function for VapourSynth, based on https://github.com/JingyunLiang/SwinIR. Dependencies NumPy PyTorch, preferably with CUDA. Note that t

Holy Wu 11 Jun 19, 2022
Half Instance Normalization Network for Image Restoration

HINet Half Instance Normalization Network for Image Restoration, based on https://github.com/megvii-model/HINet. Dependencies NumPy PyTorch, preferabl

Holy Wu 4 Jun 6, 2022
The 7th edition of NTIRE: New Trends in Image Restoration and Enhancement workshop will be held on June 2022 in conjunction with CVPR 2022.

NTIRE 2022 - Image Inpainting Challenge Important dates 2022.02.01: Release of train data (input and output images) and validation data (only input) 2

Andrés Romero 28 Jul 20, 2022
Like a cowsay but without cows!

Foxsay This is a simple program that generates pictures of a cute fox with a message. It is like a cowsay but without cows! Fox girls are better! Usag

Anastasia Kim 28 Feb 20, 2022
Old Photo Restoration (Official PyTorch Implementation)

Bringing Old Photo Back to Life (CVPR 2020 oral)

Microsoft 10.7k Aug 8, 2022
Implementation for "Exploiting Aliasing for Manga Restoration" (CVPR 2021)

[CVPR Paper](To appear) | [Project Website](To appear) | BibTex Introduction As a popular entertainment art form, manga enriches the line drawings det

null 131 Aug 9, 2022
Punctuation Restoration using Transformer Models for High-and Low-Resource Languages

Punctuation Restoration using Transformer Models This repository contins official implementation of the paper Punctuation Restoration using Transforme

Tanvirul Alam 122 Aug 12, 2022
SelfRemaster: SSL Speech Restoration

SelfRemaster: Self-Supervised Speech Restoration Official implementation of SelfRemaster: Self-Supervised Speech Restoration with Analysis-by-Synthesi

Takaaki Saeki 31 Jun 10, 2022
Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

youceF 1 Nov 12, 2021