env:

• pytorch 0.3
• python 3.5
• cuda8.0

inpainting.ipynb got this error at

Visualize : plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-e9e29a4716c2> in <module>()
      1 img_mask_var = np_to_var(img_mask_np).type(dtype)
      2 
----> 3 plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

/data/notebooks/deep-image-prior/utils/common_utils.py in plot_image_grid(images_np, nrow, factor, interpolation)
     70         interpolation: interpolation used in plt.imshow
     71     """
---> 72     grid = get_image_grid(images_np, nrow)
     73 
     74     plt.figure(figsize=(len(images_np)+factor,12+factor))

/data/notebooks/deep-image-prior/utils/common_utils.py in get_image_grid(images_np, nrow)
     57     '''Creates a grid from a list of images by concatenating them.'''
     58     images_torch = [torch.from_numpy(x) for x in images_np]
---> 59     torch_grid = torchvision.utils.make_grid(images_torch, nrow)
     60 
     61     return torch_grid.numpy()

/usr/local/lib/python3.5/dist-packages/torchvision/utils.py in make_grid(tensor, nrow, padding, normalize, range, scale_each, pad_value)
     33     # if list of tensors, convert to a 4D mini-batch Tensor
     34     if isinstance(tensor, list):
---> 35         tensor = torch.stack(tensor, dim=0)
     36 
     37     if tensor.dim() == 2:  # single image H x W

/usr/local/lib/python3.5/dist-packages/torch/functional.py in stack(sequence, dim, out)
     62     inputs = [t.unsqueeze(dim) for t in sequence]
     63     if out is None:
---> 64         return torch.cat(inputs, dim)
     65     else:
     66         return torch.cat(inputs, dim, out=out)

RuntimeError: inconsistent tensor sizes at /pytorch/torch/lib/TH/generic/THTensorMath.c:2864

I tried to run this with few other images. I am getting this error. Can you please tell me where I am going wrong? The following is the error I am getting.

RuntimeError Traceback (most recent call last) in () ----> 1 net_input = get_noise(input_depth, INPUT, (imgs['HR_pil'].size[1], imgs['HR_pil'].size[0])).type(dtype).detach() 2 3 NET_TYPE = 'skip' # UNet, ResNet 4 net = get_net(input_depth, 'skip', pad, 5 skip_n33d=128,

RuntimeError: CUDA error: out of memory

I successfully tested your inpainting algorithm for the kate.png and peppers.png on my own image (I changed only this:) elif ('kate.png' in img_path) or ('peppers.png' in img_path) or ('normal.png' in img_path)

Unfortunately on trying again with another image, I'm getting this error on the main loop:

Starting optimization with ADAM

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-13-a43cee3b5493> in <module>()
     31 
     32 p = get_params(OPT_OVER, net, net_input)
---> 33 optimize(OPTIMIZER, p, closure, LR, num_iter)

10 frames

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
   2014     return torch.batch_norm(
   2015         input, weight, bias, running_mean, running_var,
-> 2016         training, momentum, eps, torch.backends.cudnn.enabled
   2017     )
   2018 

RuntimeError: CUDA out of memory. Tried to allocate 684.00 MiB (GPU 0; 15.75 GiB total capacity; 13.88 GiB already allocated; 124.88 MiB free; 14.47 GiB reserved in total by PyTorch)

I have tried restarting the runtime twice and ran torch.cuda.empty_cache() but apparently the memory is still allocated. Would you mind telling a newbie what's going on and how to resolve this?

This is not a technical bug, but an important matter you want to address.

In the selection of example images, you have included the photo known as Lena (or Lenna).

While I'm guessing you might want to show the behavior of your algorithm on human faces, the choice of the image in this case is quite unfortunate and widely discussed as an unwanted picture in today's image processing sets.

Inspired by other projects, please consider using one of these images instead:

• Grace Hopper from the Matplotlib data set.
• Col. Eileen M. Collins from the SciKit data set.
• or any other portrait of a scientist of your own preference

Thank you for your help and addressing this report with the most careful attention.

Have a nice day

Hi, @DmitryUlyanov ,

I got the following import error, which shows it cannot find skip. I think that's path pointing issue. Would you give some suggestion to fix it? THX!

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-9abca6442a47> in <module>()
      7 
      8 import numpy as np
----> 9 from models import *
     10 
     11 import torch

/data/code/deep-image-prior/models/__init__.py in <module>()
----> 1 from skip import skip
      2 from texture_nets import get_texture_nets
      3 from resnet import ResNet
      4 # from model_pad_ds import Net
      5 from unet import UNet

ModuleNotFoundError: No module named 'skip'

Do you have the example for the flash, no flash figure? In the paper, it says you used the same parameters as the denoising example. Is this correct?

Thanks

IMO the install section should mention that pytorch needs to be compiled with CUDA support, which is quite fiddly on some macOS versions (needing to downgrade clang to compile at all, since nvcc doesn't like clang 90000).

I'm trying to test with my own image and mask and running into an issue on : plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

assert (n_channels == 3) or (n_channels == 1), "images should have 1 or 3 channels"

I've ensured that my image only has 3 channels i.e. RGB, and removed the alpha channel. But I"m guessing maybe this isn't what it's referring to?

ValueError Traceback (most recent call last)

in () 1 img_mask_var = np_to_torch(img_mask_np).type(dtype) 2 ----> 3 plot_image_grid([img_np, img_mask_np, img_mask_np*img_np], 3,11);

ValueError: operands could not be broadcast together with shapes (4,320,320) (3,320,320)

I have that strange error, does anyone know how to fix it?

Backtracking condition fails and gives an error.

Falling back to previous checkpoint.R_masked 11.899794 PSNR 11.899794 
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-16-f44ea9e7035f> in <module>()
     50 # Run
     51 p = get_params(OPT_OVER, net, net_input)
---> 52 optimize(OPTIMIZER, p, closure, LR=LR, num_iter=num_iter)

/nfs1/code/aniruddha/deep-image-prior/utils/common_utils.py in optimize(optimizer_type, parameters, closure, LR, num_iter)
    227         for j in range(num_iter):
    228             optimizer.zero_grad()
--> 229             closure()
    230             optimizer.step()
    231     else:

<ipython-input-16-f44ea9e7035f> in closure()
     25 
     26             for new_param, net_param in zip(last_net, net.parameters()):
---> 27                 net_param.copy_(new_param.cuda())
     28 
     29             return total_loss*0

RuntimeError: a leaf Variable that requires grad has been used in an in-place operation.

I could reproduce this multiple times.

There are a lot of Batch Normalization layers in the models, but the code only uses batch size of 1. What role does the BN play in the model? Have you tried training without BN and how did it perform?

There is an error on the inpaiting notebook. Because on the loss function (MSE) you use the corrupted image and also the ground truth img_var) image, so the model is always learning knowing the final image.

For superresolution.py and denoising.py

I believe this function was renamed in release 0.16 and the older compare_psnr name was removed in 0.18. You should be able to import under the new name with: from skimage.metrics import peak_signal_noise_ratio

after my "training" on restoration task, I am curious about the loss function -- mse, the parameters including both restoration generated image and ground truth, some minutes after, a function training was finished, and when I add a watermark different from further one, result as following:

last, as far as I am concerned, this algorithm is unrealistic, on the other hand, maybe I have mistaken something because I just met this project yesterday

Hi, I'm using deep prior in astronomy processing. This is an incredible process to remove random noise without introducing artefacts like a trained algorithm would do. I always have out of memory problems and Large GPU are too expansive. I have a very nice 32Gb GPU on my Mac M1 Max. Would anyone can give me some help to drop CUDA and make it work on a Mac M1 ?

One of my image that used deep prior was featured in NASA's Astronomy Picture Of the Day ! https://apod.nasa.gov/apod/ap191127.html

Hi. Can you provide some refs or papers of the skip net used for inpainting? I want to know its structure and the difference between skip-net and unet.

Hi, In inpainting.ipynb the loss is NAN from the beginning. However, if I turn off cuda:

  torch.backends.cudnn.enabled = False 
  torch.backends.cudnn.benchmark = False 
  dtype = torch.FloatTensor 

All the loss is right. I was using RTX 3070 Ti, cuda 9.0.

Why the loss turns into 'nan' when running on GPU? Please help me with this problem.