CUDA error happens occasionally during training, how can I fix it?

here is the training log and stacktrace.

(torch) ➜  DCSR git:(master) ✗ sh train.sh                                       
Making model...
Total params: 3.19M
Preparing loss function:
use_vgg: True
use_vgg: True
1.000 * L1
0.050 * contextual_ref
0.010 * contextual_hr
/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/optim/lr_scheduler.py:417: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
  "please use `get_last_lr()`.", UserWarning)
[Epoch 1]       Learning rate: 1.00e-4
/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/functional.py:3063: UserWarning: Default upsampling behavior when mode=bicubic is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode))
/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/functional.py:3103: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details. 
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed "
[400/39300]     [L1: 0.0565][contextual_ref: 0.1948][contextual_hr: 0.0653][Total: 0.3167]      49.5+1.9s
[800/39300]     [L1: 0.0414][contextual_ref: 0.1722][contextual_hr: 0.0554][Total: 0.2690]      49.1+0.1s
[1200/39300]    [L1: 0.0338][contextual_ref: 0.1618][contextual_hr: 0.0507][Total: 0.2462]      49.5+0.1s
[1600/39300]    [L1: 0.0308][contextual_ref: 0.1564][contextual_hr: 0.0480][Total: 0.2352]      49.7+0.1s
[2000/39300]    [L1: 0.0274][contextual_ref: 0.1526][contextual_hr: 0.0461][Total: 0.2260]      49.9+0.1s
[2400/39300]    [L1: 0.0253][contextual_ref: 0.1497][contextual_hr: 0.0449][Total: 0.2199]      49.9+0.1s
[2800/39300]    [L1: 0.0236][contextual_ref: 0.1473][contextual_hr: 0.0437][Total: 0.2146]      49.8+0.1s
[3200/39300]    [L1: 0.0224][contextual_ref: 0.1453][contextual_hr: 0.0428][Total: 0.2105]      49.8+0.1s
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [32,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [33,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [34,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [35,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [36,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [37,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [38,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [39,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [40,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [41,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [42,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [43,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [44,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [45,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [46,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [47,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [48,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [49,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [50,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [51,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [52,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [53,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [54,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [55,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [56,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [57,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [58,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [59,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [60,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [61,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [62,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
/opt/conda/conda-bld/pytorch_1607370120218/work/aten/src/ATen/native/cuda/ScatterGatherKernel.cu:115: operator(): block: [93,0,0], thread: [63,0,0] Assertion `idx_dim >= 0 && idx_dim < index_size && "index out of bounds"` failed.
Traceback (most recent call last):
  File "main.py", line 31, in <module>
    main()
  File "main.py", line 25, in main
    trainer.train()
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/trainer.py", line 49, in train
    sr = self.model(lr, ref)
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/__init__.py", line 100, in forward
    return self.model(x, ref,False)
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/dcsr.py", line 117, in forward
    ref_features_aligned = self.aa3(input, ref_p, index_map, ref_features1)
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/attention.py", line 115, in forward
    warpped_features = self.align(warpped_features,lr,warpped_ref)        
  File "/home/laizeqiang/miniconda3/envs/torch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/alignment.py", line 55, in forward
    p = self._get_p(affine, dtype)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/alignment.py", line 124, in _get_p
    p_n = self._get_p_n(N, dtype)
  File "/media/exthdd/laizeqiang/lzq/projects/ref-sr/related_work/DCSR/model/alignment.py", line 102, in _get_p_n
    p_n = p_n.view(1, 2*N, 1, 1).type(dtype)
RuntimeError: CUDA error: device-side assert triggered

Hi， Thanks for your tips on 4X SR! I am sparing no effort to modify the released code for 4X SR on CUFED5 because your results on CUFED5 are quite appealing. I have a few questions to ask :)

Datasets：

1. For 2X SR，the scale of LR images is the same as the ref images(2016X1512). For 4x SR, you said the scale of the ref image is 4x of that of LR images. Is it only downsampling 4X input images? Did you mean that the I Ref patch is 4X of the I LR patch? In your paper, you said the resolutions of HR and Ref are about 300×500. and which downsampling method do you use to process the input liking Figure 5 in the main paper?

2. Dose the CUFED5 datasets for training refer to the datasets with 11871 photos and testing with 126 photos?

Modify:

1. For 4X SR，I assume I need to add an extra reference features extracted from I Ref and Attention module and fusion module based on the released code.
2. when modifying the VGG19 layer from 7 to 11, it has 2 MaxPool2d. Maybe it is the input that confuses me. I feel I still need little tips to modify it successfully.

Thanks.

I try to modify the network for 4x，but I am not capable.Error message prompt：index 【】is out of bounds for dimension 2 with size 【】. I don't know how to produce the correct result

with torch.no_grad(): if coarse:

        B = lr.shape[0]
        ref_ = F.interpolate(ref, scale_factor=1/16, mode='bicubic')
        lr_ = F.interpolate(lr, scale_factor=1/8, mode='bicubic')

        i , P, r= self.select(lr_, ref_)
        
        for j in range(B):
          ref_p = ref[:, :,np.maximum((2*8*(i[j]//P)).cpu(),0):np.minimum((2*8*(i[j]//P)+2*lr.shape[2]).cpu(), ref.shape[2]), np.maximum((2*8*(i[j]%P)).cpu(),0):np.minimum((2*8*(i[j]%P)+2*lr.shape[3]).cpu(), ref.shape[3])]

      else: 
        ref_p = ref    

I run the scripts provided in the repo, but cannot reproduce 8k result as shown in the paper. Besides, the color of patches of 8K SRA is inconsistent.

sh test_8k.sh
sh test_8k_SRA.sh

result of test_8k

result of test_8k_SRA

results from the paper

Hi, Thanks for sharing your code and the excellent ideas. I am doing a large-factor RefSR project (8x or 16x SR), and I found your results on CUFED5 and your proposed dataset are quite promising. My question is how can I modify the released code for my own datasets? Looking forward to your reply. Cheers, Junhe Zhang

Hi,

I have a question on alignment.py in which affine transformation is patch-wisely applied. I am not an expert on deformable convolution, which I believe the code is based on, so please be patient with my question ;)

1. In def _get_p_n(...), why do you subtract 0.5 and add 0.6 to the range? Is it for the even-size kernel?

2. In def _get_p(...) Why do you add (self.kernel_size-1)//2+0.5 to affined transformed coordinates? Shouldn't it be (self.kernel_size-1)//2-0.5 for making coordinate range between 0~h-1?

Hi,

I've gone through the code and have some questions about the network architecture described in the paper and in the code.

1. In Figure 2 in the main paper, the network has 4 Aligned Attention(aa) modules, but the code has only 3. Is there a performance decrease when you use 4 aa modules?

2. For the aa modules defined in dcsr.py, the scale and align arguments are different based on self.flag_8k.

   • Is it appropriate to assume that the scale values are higher when self.flag_8k==True for better feature matching between features of higher resolution?
   • for an aa2 module, why does it get align==False? I can see that alignment is not necessary when scale==1, as patches become 1*1 tensors. However, for aa2 when self.flag_8k==True, why is align set to False?

3. Would you please elaborate on the intuition of having coarse==True? I have not detailly checked the patch coordinates used for the evaluation, but I assume coarse is set to True when the LR patch is outside of FOV of ref patch. When coarse==True, the DCSR model downsamples the LR and ref images with factors of 1/16 and 1/8 respectively. Is this for roughly matching the structure as those patches might not share a common context?

Thanks in advance,

Did I need modify the code in dataset? I guess this part of the code only applies to CameraFusion. I long to know the difference in dataset.py between DCSR (CUFED5) and other Refsr method .Because I originally wanted to directly apply the dataset code of other methods, but failed. @jiaxinxie97 @Tengfei-Wang

Hi, @Tengfei-Wang

First of all, I want to say thank you and ask you again.

Because the released codes are only training the overlapped FoV on CameraFusion.

So when training on CUFED5, I changed L_x to L_x = random.randrange(0, L_w - L_p + 1 - 15) for training the whole image.

When testing, I directly use the same corresponding position between the reference image and input image as a patch_ref because I think input image and reference image are very similar.

So in the second for loop, I remove those lines and select the pixel value of patch_ref[] to 4x patch_LR in the same corresponding position and set course as False.

But the result is not good, I feel my choice of patch_ref is too simple, how can I do to achieve the result in the paper when training CUFED5?

Looking forward to your reply!

Hi,

In Eq 7 in the main paper, images in the L1 term are not applied with Gaussian. However, in the code it seems like Gaussian is applied to images for the L1 loss no matter what. Is it intended?

Thanks in advance!

Hi, thank you for your code and dataset. I wonder how you collect the corresponding telephotos for wide photos? Did you manually zoom in or you have other techniques? If it is from zooming in, what is the zoom-in ratio?